Interpreting out of bag error estimate for RandomForestRegressor

Question: I am using RandomForest regressor on my data and I could see that the oob score was obtained to be 0.83. I am not sure how it came out to be like this. I mean my targets are high values in the range of 10^7. So if it's MSE then it should have been much higher. I don't understand what 0.83 signify here.
I am using python's RandomForestRegressor of the sklearn toolkit.
I do
model = RandomForestRegressor(max_depth=7, n_estimators=100, oob_score=True, n_jobs=-1)
model.fit(trainX, trainY )
Then I see model.oob_score_ and I get values like 0.83809026152005295
Comment: @Momo. I am using python's sklearn.ensemble's RandomForestRegressor. I just use the model like
Answer: In order to compare the ground truth (i.e. correct/actual) target values with estimated (i.e. predicted) target values by the random forest , scikit-learn doesn't use the MSE but $R^2$ (unlike e.g. MATLAB or (Breiman 1996b)), as you can see in the code of forest.py:
self.oob_score_ = 0.0
for k in xrange(self.n_outputs_):
    self.oob_score_ += r2_score(y[:, k], predictions[:, k])
self.oob_score_ /= self.n_outputs_

r2_score() computes the coefficient of determination aka. R2, whose best possible score is 1.0, and lower values are worse.
FYI:

What is the out of bag error in Random Forests?
What is the difference between "coefficient of determination" and "mean squared error"?
Breiman, Leo. Out-of-bag estimation. Technical report, Statistics Department, University of California Berkeley, Berkeley CA 94708, 1996b. 33, 34, 1996.


Java GeoTools - Getting CRS from SRID

Question: I might confuse between two things, but I need to do some calculation on my polygons (distance, area etc.). Using the WKBReader, I'm getting the geo:
WKBReader wkbReader = new WKBReader();
byte[] a = results.getBytes(8);
Geometry geo = wkbReader.read(results.getBytes("st_asbinary"));

But I'm not getting it in EPSG:4326. I want to use GeoTools to do the transform. How do I get the CRS from the geo object? (look at the 3 question marks)
CoordinateReferenceSystem sourceCRS = ???
CoordinateReferenceSystem targetCRS = CRS.decode("EPSG:4326");

MathTransform transform = CRS.findMathTransform(sourceCRS, targetCRS);
Geometry targetGeometry = JTS.transform(geo, transform);
Any help will be appreciated.
Answer: If a CRS value is set on the geo object then the appropriate method call is:
geo.getCoordinateReferenceSystem()

If it isn't set on the geometry then you'll need to find a suitable object to get it from.
The GeoTools Geometry CRS Tutorial has some examples (schema, world) and the GeoTools JavaDoc has a complete list of getCoordinateReferenceSystem() method declarations/implementations.
Comment: Thanks for your answer. If I want to transform the geometry to the default CSR, can I just use the following method: JTS.toGeographic(this.geo, DefaultGeographicCRS.WGS84); ???
Comment: toGeographic(...) converts into WGS84 so you would use a CRS representing your source CRS as the argument.

Java GeoTools - Getting CRS from SRID


Internet speed calculation in node.js

Question: How to calculate internet speed in server side node.js, From client side i am sending current time stamp while making the http request in query string.
client side code
<code>  var image = document.createElement("img");
      image.width = 1;
      image.height = 1;
      image.src = url+"?"+new Date().getTime();
</code>
url looks like below
<code>  http://localhost:8080/server?1369654461767
</code>
Internet speed calculation can be done using start time, end time and downloadable file size of the request.
But, My problem above request is 1x1 pixel so image size can't be determined, and i don't want one separate sever call for calculating net speed.
Any solution for calculating internet speed using start time from client and data which is present in request and response.
Any idea will be helpful.
Answer: You're not going to be able to calculate the internet speed on the server.  One critical problem is the start time is based on the client clock and the server's clock will never be in sync enough for the purpose of calculating speed.
To calculate speed you need a full round-trip where you can capture start and end time accurately from the same machine--both on the client.  You also need to send something more significant than a 1px image.  The smaller the payload, the less accurate the calculation.
Typically speed is calculated by requested several large responses of known sizes.

Internet speed calculation in node.js


Locale specific javascript files with assets pipeline

Question: So what I want to know is whether the following is possible. I have a website where we internationalise using the standard I18n gem (with translations in config/locales/*.yml). At the moment we've relied on a variety of (pretty inelegant) fixes to get javascript to pick up translated words (mostly by use of the HTML5 data- attributes). Now that we've upgraded our site to Rails 3.1, and we're liking the new asset pipeline stuff, it'd be great if we could internationalize up our js files something like the following:
app/assets/jsfile.js.erb
alert(<%= I18n.t('javascript.key.name') %>)

and then get the assets pipeline to iterate over the file to generate public/assets/jsfile.en.js, public/assets/jsfile.de.js etc etc (I'm guessing that a locale aware javascripts_include_tag would also be useful)
Is anyone aware of any examples/gems/plugins that could do this kind of job, or is it time to crack out the IDE and start coding up something myself?
Answer: I had the same problem and in the end I decided to implement something.
I added to assets.rake some tasks which create the localized files specified in
config.assets.localized_precompile, for example:
config.assets.localized_precompile = ['application-it.js', 'application-fr.js', 'application-de.js', 'application-en.js']

Then I'm using
=javascript_include_tag "application-#{I18n.locale}"

And application-xx.js are all like:
//= require application

I can share more details if you are interested
Comment: Hmm interesting. That looks broadly like the kind of the solution we're looking for. Thanks for sharing the concept.
Comment: I had the authorization to extract a gem: https://github.com/simplificator/asset_pipeline_i18n . Hth
Comment: I like this, seems to work in rails 4 too! But need to add you new js to `Rails.application.config.assets.precompile += `


How can I stop JavaScript from being cached during development?

Question: I have a Drupal 8.00-beta11 site and am trying to dynamically add JS to a page, so that I don't have to clear the cache every time I edit the JS file. I have a custom module (lets call it my_module) in drupal_root/modules. My javascript file is in drupal_root/modules/my_module/js/my_js.js.
I have a my_module.libraries.yml file as follows:
my_module:
  version: 1.x
  js:
    js/my_js.js: {}
  dependencies:
    - core/jquery
    - core/jquery.ui

In my my_module.info.yml I have:
name: My Module
type: module
description: ...
core: 8.x
libraries:
  - my_module/my_module
package: Custom
version: 0.1

This loads the JS succesfully, but I need to clear the cache everytime after I edit it to see the changes.
In order to avoid doing this, I added the following to my buildForm function:
$form['#attached']['js'] = array(
  drupal_get_path('module', 'my_module') . 'js/my_js.js',
);

$form['#attached']['library'][] = 'my_module/my_module';

I also placed the following in drupal_root/modules/my_module/my_module.module:
<?php

  function grid_tools_page_attachments(array &$attachments) {
    $attachments['#attached']['library'][] = 'my_module/my_module';
  }

  function grid_tools_page_build(array &$attachments) {
    $attachments['#attached']['library'][] = 'my_module/my_module';
    $path = drupal_get_path('module', 'my_module');
  // Add JavaScript/CSS assets to all pages.
  // @see drupal_process_attached()
    $attachments['#attached']['js'][$path . '/js/my_js.js'];
  }

?>

However, none of this has worked and I still need to clear the cache to see my changes. How do I avoid having to clear the cache every time?
Answer: I'm assuming you have to clear caches because your JS file is being aggregated? Can't think of a different reason.
That has absolutely nothing to do with how you add your JS file (Most of what you tried shouldn't even work, use #attached][library).
Instead, it depends on the separate settings to aggregate JS or not. You can find them on the Performance settings page. Aggregation is also automatically disabled if you use the provided sites/example.settings.local.php as your settings.local.php, which I strongly recommend when doing development, also for the other things it does like enable verbose error logging and disabling render caching (but keep in mind to always test your code with render caching too).
Answer: It won't work because you are using libraries and they will get aggregated and cached. To have really dynamic JS I would suggest loading the content of the JS file into header within script tags in hook_page_attachments_alter(). See the system_page_attachment() for more information on how you could do it.
But this is really bad idea. Instead I would suggest you clear the CSS/JS cache  by running the _drupal_flush_css_js() if you don' want to flush all caches.


How to open (and save) all embedded documents in a Word document?

Question: I'm looking to write VBA to save out embedded files in a Word document.  But I'm having issues just opening them:
Sub Extract()

    Dim num as Integer

    Dim numObjects As Integer
    numObjects = ActiveDocument.InlineShapes.count

    MsgBox numObjects  ' prints "11"

    For num = 1 To numObjects
        If ActiveDocument.InlineShapes(num).Type = 1 Then
            'it's an embedded OLE type so open it.
            ActiveDocument.InlineShapes(num).OLEFormat.Open

            'Works for the first one but errors 5941 (the requested
            '  member of the collection does not exist)
        End If
    Next num

End Sub

This code opens the first embedded file if it's not already open.  It errors on the next one.
Or, if the first file is already open, the macro just seems to do nothing.
Any hints?  (I'm doing this with Word 2010.)
Answer: The answer seems to be quite easy- after you open first embedded file it becomes active one and next when you try to open the next embedded object you refers to active document, not to desired one. Try with object variable in this way:
Sub Extract()

    Dim num as Integer
    Dim AD as document
    Set AD = activedocument

    Dim numObjects As Integer
    numObjects = AD.InlineShapes.count

    MsgBox numObjects  ' prints "11"

    For num = 1 To numObjects
        If AD.InlineShapes(num).Type = 1 Then
            'it's an embedded OLE type so open it.
            AD.InlineShapes(num).OLEFormat.Open

        End If
    Next num

End Sub


CSS checkbox styling won't work

Question: I'm having a bit of a problem here. I read a ton of topics on this subject, but in won't work for me...
I'm making a theme for wp atm but I'm stuck on CSS checkbox styling, This is my CSS:
 input[type=checkbox] {
    opacity: 0;
  }

  input[type=checkbox] + label
   {
    background-image: url('images/main-sprite.png') no-repeat 0 -311px;
    height: 20px;
    width: 20px;
       display:inline-block;
       padding: 0 0 0 0px;
   }

   input[type=checkbox]:checked + label
    {
            background-image: url('images/main-sprite.png') no-repeat 0 -357px;
    height: 20px;
    width: 20px;
        width: 16px;
        display:inline-block;

        padding: 0 0 0 0px;
    }
.lcheckbox {
        position: relative;
    top: 43px;
    right: 200px;
}

And ofcourse my HTML:
<input class='lcheckbox' type="checkbox" />Remember me

So, is there anybody who can help me?
Thanks
Okay, I've got the first thing fixed, but now wheb I click nothing happens, and the label text is like:
Remember *Here's a enter, so we continue on the next line*
Me
Comment: where's the `label` in your HTML? also it would be helpful if you describe what the issue is and if you provide a reproducible example in a [jsfiddle](http://jsfiddle.net)
Comment: you have `input + checkbox`. Isn't that improper HTML? Labels are supposed to surround checkboxes, especially since if they don't won't it not manipulate the checkbox if you click on the label text?
Comment: @Leeish Could u make that more simple for me? I'm Dutch, so my English is not that good.
Comment: and also it should be `input[type='checkbox']`
Comment: a label should surround a checkbox, not come after it I think.``
Comment: @Mr Lister, isn't the `'` or `"` around checkbox required in the css?
Comment: @MrLister yeah you right, my bad ;) "This is referred to as an adjacent selector. It will select only the element that is immediately preceded by the former element. In this case, only the first paragraph after each ul will have red text."
Comment: @Leeish only if the value contains spaces or other non-word characters.
Comment: @user3135286 Re your edit: The label text gets put in two lines, because the label is not wide enough to hold the whole line. It's only 20 pixels across! So only the left part is clickable, the part where the sprite is. You do see the sprite, don't you? Otherwise, check if main-sprint.pg exists.
Answer:  works on an input that has the id checkbox.
Since your input doesn't have an id, it won't work.
Solution: give the input an id.
<input class='lcheckbox' type="checkbox" id="checkbox" />
                                         ^^^^^^^^^^^^^

(or any other id, as long as it matches the label's for attribute.)

CSS checkbox styling won't work


is the distinguishedName order in ActiveDirectory strict?

Question: I'm trying to convert the distinguishedName of a Domain into the address for the domain and am wondering if the order is strict. They all seem to be (after using the Active Directory explorer http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx to explore ActiveDirectory...). I just haven't found anywhere which states the order is strict...
so, I'm hoping that:
DC=safety,DC=company,DC=com

is always equal to
safety.company.com

or could safety.company.com be stored in ActiveDirectory with any of the following etc:
DC=com,DC=company,DC=safety
DC=company,DC=com,DC=safety

Ultimately when passed the DC=...string, I want to be able to do the following to store the address:
Dim ADPath As String = "DC=safety,DC=company,DC=com"
Dim DomainAddress As String = ADPath.Replace(",DC=",".").Replace("DC=","")

Answer: Not the easiest text to read (for me anyway) but this MSDN page does describe that the path from left to right is a hierarchical path that uniquely describes the object.
"A DN is a sequence of relative distinguished names (RDN) connected by commas."
source: http://msdn.microsoft.com/en-us/library/aa366101%28VS.85%29.aspx


How to switch from Trial to Licenced CAL Windows Server 2008

Question: I have finaly purchased my RDS CAL licences and now need to switch user from Trial to Licence but even if I remove from registry trial info from server or client the client still get Trial 120 days.
My server show 20 licences installed and avalable and 0 in use.
Comment: Possible duplicate of [Can you help me with my software licensing issue?](https://serverfault.com/questions/215405/can-you-help-me-with-my-software-licensing-issue)
Comment: Last time I tried to activate RDP licenses for Windows Server 2008R2, the person on the phone said they couldn't process it...
Answer: Please make sure your correct type of CAL is installed.
If you install device CAL and your TS is set to issue User CAL, you will end up in the trial bucket.
tsconfig.msc, and validate in what mode your RDS server issue the CAL.


Net group command

Question: I'm working on an automation project, and I need to add members to a specific group using a command-line approach. Currently, I'm using the net group command like this:
<code>net group "group name" user /add /domain
</code>
However, the group name can change, which isn't ideal for automation. Is there a way to use the group email address instead of the group name in an equivalent command line? The email address remains constant and doesn't change !! can someone please help me through this
thanks alot :)
Answer: Use Powershell for this.
Read in the group and users in a variable first. The users can either be hardcoded or like for the group - we can apply a filter, query AD and add the returned user(s) to the group.
<code>$Group = Get-ADGroup -Filter "mail -eq 'email@example.com'"
$Users = Get-ADUser -Filter [...]

Add-ADGroupMember -Identity $Group -Members $Users
</code>

Net group command


Disable scp password prompt

Question: I have a user ID set up on a server that doesn't require a password.  I'd like to be able to use scp to transfer a file from it.  My problem is scp keeps asking for my password even though there isn't one; I can telnet to the server and log on without the password.  Is there any option (-o) I can specify to disable the password prompt?  Using keys is not an option.
Comment: Stackoverflow is for help writing software. You should consider asking on http://superuser.com/ instead.
Answer: I'm no expert, but I'm guessing you might have to set PermitEmptyPasswords to Yes in /etc/ssh/sshd_config as per the instructions here.

Disable scp password prompt


Other ways to practice singing

Question: I'm new to singing and music and I'd like to be a great singer one day but I can't always sing; that wouldn't be healthy for my voice.
(Also at some point if I have to sing 'Holy' again I might just become unsanctified.) Are there other computer guided ways to practice singing that are low impact? I know about ear training but I'm a baritone and most software only supports tenor and bass.
Answer: If singing feels "high-impact," there might be some part of your vocal instrument that you are "over-handling."  Singing should feel "low-impact" when done with proper technique.  You should feel the resonance on your lips far more than anywhere internal.  Aim to have speaking and singing feel the same.  A voice coach is highly recommended.
Answer: Hey I'm also pretty new to singing and have aspirations of excellence.  I've learned a few things and they have helped a lot.  I would say number one before technique is to do it everyday.  Your vocal chords are muscles and probably have not been properly exercised if you didn't sing when you were young like me.
To deal with keeping your chords from wearing out is a key awareness of your body.  Try singing low and focus on how it feels.  You should feel a resonance in your chest.  Now try raising the pitch and you should feel the resonance move higher and higher into your head.  There's actually a nasal cavity right behind your nose that goes off when you get really high.
The key is not to strain your throat or neck muscles as you raise your pitch, or going lower for that matter, but it is much more common for people to strain on the high notes.  You can hear a stained note and it doesn't sound good.  You want to support your pitch with proper head/chest resonance.  It is exactly the same as the "lift with your knees, not your back" technique.  You want your body to support it easily and not injure yourself.  If you master this people will be amazed and how powerful your voice can be without too much effort.  Don't push your range too much without warming up and easing into it.  That is another injury pitfall.
One last thing because I think it is important for what I just said.  If you take a breath you probably puff out your chest because that is the normal thing to do.  You can actually use your lower diaphragm to fill your lungs at the belly first.  "why the zeus would I do that?" you ask?  Well, if your fill your chest first you cut off that lower part of your lungs and if you fill up the bottom first you make a solid foundation of air.  A pillar of vocal strength.  There's no need to take an overly large breath because that just creates muscle tension which as we now know is bad.  But having a good supply of ammunition always helps.
That was a lot but if your like me you can't afford vocal techniques right now and the teacher selection is not that great where you live anyway, so I wanted to share a lot.  I bought a keyboard for warming up my voice.  I sing up and down with the keys in lots of silly ways until my range expands.  There's lots of the scales to do.  I sing a chord to the fifth most of the time and then move up or down a semi-tone and repeat.  Oh ya!  At first focus on one technique at a time.  If you try to do all these things at once perfectly it will drive you crazy!
Happy singing!
Answer: Can't understand why singing wouldn't be healthy for a voice.If one can talk, shout etc., then one can sing.If you're baritone, surely the software can be made to change key to produce a tessitura that matches your range.'Holy' - Try other two syllable words. There are plenty to choose from - words don't HAVE to make sense. (Sometimes they don't anyhow).Hum - this involves keeping to the key, without straining parts of the mouth. Scat sing, over any track you like.There are many solutions, it's surprising no others are forthcoming.
Comment: In order to annihilate one's voice one would have to over-use it for many hours.Bodies have various ways to tell their owners they've had enough and are close to the danger zone.Pain and discomfort being the fore-runners.
Comment: It's possible to destroy your your voice by overuse
Comment: www.nahealth.com/NewsAndEvents/HealthEducationArticles/PhysicalTherapy/VoiceSafety
Comment: Cumulative effects happen too

Other ways to practice singing


How to compare merge two arrays in jquery

Question: I am having two arrays:
existing_names = username1, username2, username3;

new_names = username1, username4, username5;

My output Should be:
new_names = username1, username2, username3, username4, username5;

How can i able to do this using jquery...
Comment: You can do this with JavaScript... Look at this question

http://stackoverflow.com/questions/1584370/how-to-merge-two-arrays-in-javascript
Answer: Use snook's OC function and its fairly easy with pure Javascript. Could possibly be done shorter with jQuery, but always fun to do without :)
DEMO: http://jsfiddle.net/wesbos/MvurR/
var existing_names = ["username1", "username2", "username3"];

var new_names = ["username1", "username4", "username5"];

/* http://snook.ca/archives/javascript/testing_for_a_v */

function oc(a)
{
  var o = {};
  for(var i=0;i<a.length;i++)
  {
    o[a[i]]='';
  }
  return o;
}

for (var i=0; i < existing_names.length; i++) {
    if (!(existing_names[i] in oc(new_names))) {
        new_names.push(existing_names[i]);
    }
}

console.log(new_names);

Comment: Are you really creating an object of all new_names on every iteration of the loop?  That's not very efficient.  You really ought to move the oc function call before the loop.
Answer: http://jsfiddle.net/AYgNW/3/
var existing_names = ['username1','username2','username3'];
var new_names = ['username1','username4','username5'];

function mergeArrays( a1, a2 ) {

    var len1 = a1.length;
    var len2 = a2.length;
    var found = false;

    for ( var x = 0; x < len2; x += 1 ) {

        found = false;

        for ( var y = 0; y < len1; y += 1 ) {
             if ( a1[y] === a2[x] ) {
                found = true;
                break;
            }
        }

        if ( ! found ) {
            a1.push( a2[x] );
        }

    }

    return a1;

}

// merge arrays
new_names = mergeArrays( new_names, existing_names );

// sort the resulting array
new_names.sort();

for ( var x = 0; x < new_names.length; x += 1 ) {
    document.write(new_names[x] + ', ');
}


Coding factors with non-numeric settings

Question: For my role as a TA I am assisting the instructor in developing curriculum based on DOE and statistics.
We have a set of failure strengths (the output/dependent variable) for several glass rods. The independent variables are diameter and surface finish. While there is numeric information for the diameter, the surface finish is marked as A for abraded and S for smooth. While this could be coded as -1 and +1, I have no idea how to interpret the coefficients of the model once a linear regression has been performed.
How can we code the data and subsequently use the model as a predictive tool?
Comment: Are diameter and surface finish really dependent variables or are they rather the input/independent variables?
Comment: Faulkner - I took the liberty of editing to make the second "dependent" to read "independent" (and fixed another typo while there). Please check that it says what you intend.
Answer: There are any number of ways to validly code the "surface finish" variable. The most common way would be to code them as 0 and 1. That way the coefficient in a main effects model (strength ~ diameter + finish) would estimate the average difference in strength for the two finishes, given the same diameter. The one coded as 0 would be the "baseline" category. The coefficient for the 0-1 dummy in the regression would represent the shift from the baseline mean to get the other mean.
If you code as (-1,1), then the coefficient represents an estimate of half the difference in population means (and the meaning of the intercept also changes).
To use in predictions, it doesn't matter how you code; you just multiply the coefficient by the coding you used.
I'll see if I can mock up an example. If you have a small sample data set (say a dozen observations) I could use that.
Okay, here's a small example (not only the data but also the units are made up):
strength diameter finish
32.13075       17      A
40.15048       16      S
20.10630       12      A
26.13942       15      A
59.98352       14      A
25.92413       13      A
79.81273       14      S
70.15717       24      A
56.01118       20      A
22.08519        7      S
93.19808       24      S
91.86548       24      A
34.09182       13      A
83.82144       18      S
40.19491       17      S
66.01798       22      S
46.06970       19      A
51.92729       20      A
19.97026       15      A
56.13245       18      A
54.15978       15      S
28.03752       12      A
75.99303       18      S
 2.13246        4      A
25.98175       14      A

Now if we code a dummy "finish.S" as A=0, S=1
then the model can be with IVs diameter and finish.S
We can then fit a regression:
Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -19.2460     9.9774  -1.929  0.06674
diameter      3.7583     0.5885   6.387    2e-06
finish.S     17.9057     5.8642   3.053  0.00583

Residual standard error: 14.02 on 22 degrees of freedom
Multiple R-squared:  0.7109,    Adjusted R-squared:  0.6846
F-statistic: 27.04 on 2 and 22 DF,  p-value: 1.181e-06

The fitted equation is
strength = -19.25 + 3.7583 diameter + 17.91 finish.S

where finish.S = 1 if finish="S" and is 0 otherwise.
Here's a plot of the two fitted lines:

on the other hand, if you coded as A=-1 and S=1:
Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -10.2932    10.0415  -1.025  0.31647
diameter      3.7583     0.5885   6.387    2e-06
finish.1      8.9528     2.9321   3.053  0.00583

Residual standard error: 14.02 on 22 degrees of freedom
Multiple R-squared:  0.7109,    Adjusted R-squared:  0.6846
F-statistic: 27.04 on 2 and 22 DF,  p-value: 1.181e-06

The fitted equation is
strength = -10.29 + 3.7583 diameter + 8.95 finish.1

where finish.1 either has the value 1 or -1.
So the line for "A":
strength = (-10.29 - 8.95)   + 3.7583 diameter

and the line for "S":
strength = (-10.29 + 8.95) + 3.7583 diameter

My rounding error aside, you end up with the same two fitted lines (the same predictions) with either coding.


Hyperbolic contour integration

Question: $$
\int_{-\infty}^\infty\frac{x\cos(ux)}{\sinh(x/2)}dx
$$
If I'm correct, there is a pole at $z = i\pi$ ?
How can I evaluate this integral?
Comment: If you want a quick guide to formatting math on this site, you can find one [right here](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference). You could also take a look at the source for your question post (click the "edit" button) to see how I did what I did.
Comment: @xpaul $\sinh$ is not $\sin$.
Answer: Note that
$$ \int_{-\infty}^\infty\frac{x\cos(ux)}{\sinh(x/2)}dx=2\int_{-\infty}^\infty\frac{x\cos(2ux)}{\sinh(x)}dx.$$
Let $f(z)=\frac{z\cos(2uz)}{\sinh(z }$.
Choose the rectangle $\gamma$: $z=-R$ to $z=R$ in $x$-axis, $z=R$ to $z=R+\pi i$ in the line $x=R$,  $z=-R+\pi i$ to $z=-\varepsilon+\pi i$ and $z=\varepsilon+\pi$ $z=R+\pi i$ in the line $y=\pi i$, $z=-R+\pi i$ to $z=-R$ in the line $x=-R$, and a small semicircle $C_\varepsilon$ below the pole $z=\pi i$  with radius $\varepsilon$. Then $f(z)$ is analytic inside $\gamma$. Note that he contribution from the short sides of the rectangle is zero since $\sinh$ grows exponentially as $x\to\pm\infty$. Then one has
$$ \int_{-\infty}^\infty f(x)dx  - \int_{\varepsilon}^\infty f(x+i\pi)dx -\int_{C_\varepsilon} f(x)dz - \int_{-\infty}^{-\varepsilon} f(x+i\pi)dx =0.$$
Note that $z=\pi i+\varepsilon e^{it}$ on $C_\varepsilon$, $t\in[\pi,2\pi]$. So
$$ \int_\pi^{2\pi} \frac{(\varepsilon e^{it}+i\pi)\cos[2u(\varepsilon e^{it}+i\pi)]}{\sinh(\varepsilon e^{it}+i\pi)}i\varepsilon e^{it}dt \to \int_{\pi}^{2\pi} \pi\cosh(2\pi u) dt = \pi^2\cosh(2u\pi)$$
as $\varepsilon\to0$. Here $\sinh(x+i\pi) \approx -x$ near $x=\pi i$ is used.
Thus we have $$ \pi^2\cosh(2u\pi) = \int_{-\infty}^\infty f(x)dx - \lim_{\epsilon\to 0}\left(\int_{-\infty}^{-\epsilon} f(x+i\pi)dx+\int_{\epsilon}^\infty f(x+i\pi)dx\right). $$
Using $$\sinh(x+i\pi) = -\sinh(x),\cos(xi)=\cosh(x),\sin(xi)=i\sinh(x)$$ and the principle integral, it is easy to see
\begin{eqnarray*}
&&\int_{-\infty}^{-\epsilon} f(x+i\pi)dx+\int_{\epsilon}^\infty f(x+i\pi)dx\\
&=&-\int_{-\infty}^{-\epsilon}\frac{(x+\pi i)\cos[2u(x+\pi i)]}{\sinh(x)}dx-\int_{\epsilon}^\infty \frac{(x+\pi i)\cos[2u(x+\pi i)]}{\sinh(x)}dx\\
&=&-\bigg(\int_{-\infty}^{-\epsilon}+\int_{\epsilon}^\infty\bigg)\frac{(x+\pi i)[\cos(2ux)\cosh(2\pi u)-i\sin(2ux)\sinh(2\pi u)]}{\sinh(x)}dx\\
&=&-\bigg(\int_{-\infty}^{-\epsilon}+\int_{\epsilon}^\infty\bigg)\frac{x\cos(2ux)\cosh(2\pi u)+\pi\sin(2ux)\sinh(2\pi u)}{\sinh(x)}dx\\
&&-i\bigg(\int_{-\infty}^{-\epsilon}+\int_{\epsilon}^\infty\bigg)\frac{\pi \cos(2ux)\cosh(2\pi u)-x\sin(2ux)\sinh(2\pi u)}{\sinh(x)}dx\\
&=&-\bigg(\int_{-\infty}^{-\epsilon}+\int_{\epsilon}^\infty\bigg)\frac{x\cos(2ux)\cosh(2\pi u)}{\sinh(x)}dx\\
&\to&-\cosh(2\pi u)\int_{-\infty}^\infty f(x)dx
\end{eqnarray*}
as $\varepsilon\to0$.
$$ \int_{-\infty}^\infty f(x)dx = \frac{\pi^2\cosh(2\pi u)}{1+\cosh(2\pi u)}.$$
So
$$
\int_{-\infty}^\infty\frac{x\cos(ux)}{\sinh(x/2)}dx=\frac{2\pi^2\cosh(2\pi u)}{1+\cosh(2\pi u)}.
$$
Answer: $\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\mc}[1]{\mathcal{#1}}
 \newcommand{\mrm}[1]{\mathrm{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$
$$
\begin{array}{c}\hline
\quad Zeros\ \mbox{of}\ \sinh\pars{x \over 2}\ \mbox{are given by}\ \braces{z_{n} = 2n\pi\ic\ \mid\ n \in \mathbb{Z}}\quad
\\[2mm]
\mbox{So,}\ \color{red}{\large \pi\ic}\ \mathbf{isn't}\
\mbox{a pole of the integrand.}
\\ \hline
\end{array}
$$

An alternative:
\begin{align}
&\bbox[10px,#ffd]{\int_{-\infty}^{\infty}{x\cos\pars{ux} \over \sinh\pars{x/2}}\,\dd x} =
2\int_{0}^{\infty}
{x\pars{\expo{\ic\verts{u}x} + \expo{-\ic\verts{u}x}}/2 \over \pars{\expo{x/2} - \expo{-x/2}}/2}\,\dd x
\\[5mm] = &\
2\int_{0}^{\infty}
{x\bracks{\expo{-\pars{1/2 - \ic\verts{u}}x} +
\expo{-\pars{1/2 + \ic\verts{u}}x}} \over 1 - \expo{-x}}\,\dd x
\\[5mm] \stackrel{t\ =\ \expo{-x}}{=}\,\,\,&
2\int_{1}^{0}{\bracks{-\ln\pars{t}}
\pars{t^{1/2 - \ic\verts{u}} + t^{1/2 + \ic\verts{u}}} \over 1 - t}\,\pars{-\,{\dd t \over t}}
\\[5mm] = &
-2\,\int_{0}^{1}{\ln\pars{t}
\pars{t^{-1/2 - \ic\verts{u}} + t^{-1/2 + \ic\verts{u}}} \over 1 - t}\,\dd t
\\[5mm] = &\
-2\,\lim_{\nu \to 0}\partiald{}{\nu}\int_{0}^{1}{\pars{t^{\nu} - 1}
\pars{t^{-1/2 - \ic\verts{u}} + t^{-1/2 + \ic\verts{u}}} \over 1 - t}\,\dd t
\\[5mm] = &\
2\,\lim_{\nu \to 0}\partiald{}{\nu}\bracks{%
\int_{0}^{1}{1 - t^{\nu - 1/2 - \ic\verts{u}} \over 1 - t}\,\dd t +
\int_{0}^{1}{1 - t^{\nu - 1/2 + \ic\verts{u}} \over 1 - t}\,\dd t}
\\[5mm] = &\
2\,\lim_{\nu \to 0}\partiald{}{\nu}\bracks{%
\Psi\pars{\nu + {1 \over 2} - \ic\verts{\mu}} +
\Psi\pars{\nu + {1 \over 2} + \ic\verts{\mu}}}
\end{align}

where $\ds{\Psi}$ is the Digamma Function.

Then,
\begin{align}
&\bbox[10px,#ffd]{\int_{-\infty}^{\infty}{x\cos\pars{ux} \over \sinh\pars{x/2}}\,\dd x} =
2\bracks{\Psi'\pars{{1 \over 2} - \ic\verts{\mu}} +
\Psi'\pars{{1 \over 2} + \ic\verts{\mu}}}
\\[5mm] = &\
\bbx{2\pi^{2}\,\mrm{sech}^{2}\pars{\pi u}}
\quad
\pars{\begin{array}{l}
\mbox{In the last line, I used the}
\\[2mm]
Euler\ Reflection\ Formula
\end{array}}
\end{align}
Answer: We have
$$\frac {\sin u x} {\sinh(x/2)} =
\frac {i (e^{-i u x} - e^{i u x})} {e^{x/2} - e^{-x/2}}, \\
\operatorname*{Res}_{x = 2 \pi i k} \frac 1 {e^{x/2} - e^{-x/2}} = (-1)^k.$$
Deform the contour of integration to $(-\infty, -\epsilon] \cup C_\epsilon \cup [\epsilon, \infty)$, where $C_\epsilon$ is a small half-circle in the lower half-plane centered at the origin. For $u > 0$, evaluate the integral containing $e^{i u x}$ ($e^{-i u x}$) by closing the contour into the upper (lower) half-plane:
$$I(u) = \int_{\mathbb R} \frac {\sin u x} {\sinh(x/2)} dx =
2 \pi i \left( -i - 2 i \sum_{k \geq 1}
 (-1)^k e^{-2 \pi u k} \right) = \\
2 \pi - \frac {4 \pi} {1 + e^{2 \pi u}} =
2 \pi \tanh \pi u.$$
The same holds for $u < 0$ because the integrand is an odd function of $u$. Then
$$\int_{\mathbb R} \frac {x \cos u x} {\sinh(x/2)} dx = I'(u).$$

Hyperbolic contour integration


How to select Development team for iOS app created in Flutter

Question: I am trying to deploy an iOS app to Device using Google's Flutter, but I get this error on the command line:
Building an iOS app requires a selected Development Team with a Provisioning Profile
Please ensure that a Development Team is selected by:
  1- Opening the Flutter project's Xcode target with
       open ios/Runner.xcworkspace
  2- Select the 'Runner' project in the navigator then the 'Runner' target
     in the project settings
  3- In the 'General' tab, make sure a 'Development Team' is selected
For more information, please visit:
  https://flutter.io/setup/#deploy-to-ios-devices
Or run on an iOS simulator

Where and how do I do this? There is no "ios/Runner.xcworkspace" in the project. I know the installed Certificates and Provisioning profile work on my Mac, because I created and installed an app on my iOS device using Appcelerator, and it found everything I needed, and ran perfectly on device.
Answer: Select Runner, and navigate to Signing & Capabilities, under Signing, check Automatically manage signing, after that choose a Team from the list, and you're done.
Comment: "Runnner" Button does neither exist in Android Studio nor Xcode :(
Comment: One trick for flutter devs: you need to open the iOS folder in XCode. This is a subdirectory under the main project folder
Answer: You have to sign in with your appleID at 'signing'. After that you can select your 'personal team'.
Answer: open ios/Runner.xcodeproj and follow the same instructions should work?  Runner.xcodeworkspace is a relatively new addition to the template, created to allow Flutter projects to integrate better with CocoaPods.
If that doesn't work, please let us know.
Comment: Could you please elaborate your first sentence. "open ios/Runner.xcodeproj and follow the same instructions should work?" what does that mean?


Understanding Proof of Eisenstein's Criterion

Question: The formulation of Eisenstein's criterion in my text is:

Let $a(x) = a_0 + a_1x+\cdots+a_nx^n$ be a polynomial with integer coefficients. Suppose there is a prime number $p$ which divides every coefficient of $a(x)$ except the leading coefficient $a_n$; suppose $p$ does not divide $a_n$ and $p^2$ does not divide $a_0$. Then $a(x)$ is irreducible over $\mathbb{Q}$.

I want to understand the proof of this given on Wikipedia. The proof outline is the following. Assume by contradiction that a polynomial $Q=q_n x^n + \cdots + q_0$ satisfying the conditions for some $p$  is reducible such that $Q=G*H=(g_mx^m+\cdots+g_0)(h_kx^k+\cdots+h_0)$. Then send both sides of $Q=G*H$ to $\mathbb{Z}_p[x]$ (using the homomorphism induced by the natural mapping from $\mathbb{Z} \rightarrow \mathbb{Z_p})$ and you are necessarily left with $\bar{q_n}x^n = (\bar{g_m}x^
m)( \bar{h_k}x^k)$. This implies that $g_0$ and $h_0$ are multiples of $p$, contradicting the assumption that $p^2\nmid q_0$.
Why is the bolded statement above true? How can I prove that there aren't other polynomials $G',H' \in \mathbb{Z}_p[x]$ with more terms than just the leading terms $\bar{g_m}x^m, \bar{h_k}x^k$, respectively, such that $\bar{q_n}x^n = G'*H'$ ?
Comment: $\mathbb{Z}_{p}[x]$ is a unique factorization domain, so the situation you describe in particular is impossible.
Comment: Because you know that $p$ divides each $q_i$ for $i \leq n-1$, so only the leading coefficient on the left can remain, and if you expand the RHS, only that term will have the matching degree. I am not sure what your second question is asking, since there are of course many polynomials $G,H$ such that $GH = \bar q_nx^n$
Answer: Let $i$ be the smallest index such that $\bar{g}_i\neq 0$, and $j$ the smallest index such that $\bar{h}_j\neq 0$.
Then the smallest degree term of $\overline{G}\cdot\overline{H}$ is $\bar{g}_i\bar{h}_jx^{i+j}$. On the other hand, we know that $\overline{G}\cdot\overline{H}=\bar{q}_nx^n$, hence $i+j=n$. Since $i\leq m$ and $j\leq k$, and $m+k=n$, this is only possible if $i=m$ and $j=k$. Therefore all the lower degree terms must be zero, as claimed.


Bitstrings and set of solutions

Question: There is something about counting bitstrings and the format of the solutions I didn't really understand yet.
Given a bitstring problem that asks to elementarily count how many bitstrings of length 36 there are that contain exactly 5 $\{1\}$ in the first 10 posititions and the $\{10100101\}$ substring in the last 20 positions, finding it out is pretty straightforward.
Since the first 10 positions need to contain exactly 5 $\{1\}$, you find out the number of all the possible combinations containing maximum five ones:
$${10 \choose 5}$$
We have no information about the next 16 positions, so that is $2^{16}$ more combinations, and thus far we have:
$${10 \choose 5} \cdot 2^{16}$$
many combinations.
Now the substring:

there are $\{10100101\} \cdot 2^{12}$ bitstrings with no overlapping; $20 - 8 + 1 = 13 \rightarrow {13 \choose 1} \cdot 2^{12}$
there are $\{1010010100101\} \cdot 2^{7}$ bitstrings with single overlapping, same goes as you overlap the starting $1$ and the one at the end, so ${8 \choose 1} \cdot 2^{7}, {6 \choose 1} \cdot 2^{5}$
there are $2\cdot \{10100101\} \cdot 2^{4}$ bitstrings as the substring repeats itself once, so ${6 \choose 2} \cdot 2^{4}$
there are $\{101001010010100101\} \cdot 2^{2}$ bitstrings with double overlapping, and considering also all the other possibilities you have ${3 \choose 1} \cdot 2^{2}, {1 \choose 1} \cdot 2^{0}, {1 \choose 1} \cdot 2^{0}$

Since $\{10100101\} \cdot 2^{12}$ "spans" all the other combinations below and are thus included within it, shouldn't we remove all the overlapping and repeating substrings from the universe $\{10100101\} \cdot 2^{12}$? Given the number of substrings above, how should I then build the solution?
Comment: The number of binary strings of length $10$ with exactly five $1's$ is just $\binom {10}5$...there shouldn't be a multiplier of $2^5$.
Comment: My bad, fixed, thank you.
Answer: In order to calculate the number of binary strings of length $20$ containing the substring $10100101$ pretty much of all the hard work is already done. We just have to use the inclusion-exclusion principle and put all the intermediate results together.

The number of wanted strings is
  \begin{align*}
&\binom{13}{1}2^{12}-\left(\binom{8}{1}2^7+\binom{6}{1}2^5+\binom{6}{2}2^4\right)
+\left(\binom{3}{1}2^2+\binom{1}{1}2^0+\binom{1}{1}2^0\right)\\
&\qquad=53\,248-(1024+192+240)+(12+1+1)\\
&\qquad\,\,\color{blue}{=51\,806}
\end{align*}
  which counts the number of binary strings of length $20$

containing the substring $10100101$ at least once
minus the number of strings  where the substring occurs at least twice, including overlaps.
Since we have subtraced strings where the substring occurs at least three times (here in form of overlaps only) too often we have to add them for compensation.

We can check the result by applying another approach called the Goulden-Jackson Cluster Method.

We consider the set words of length $n\geq 0$ built from an alphabet $$\mathcal{V}=\{0,1\}$$ and the set $B=\{010100101\}$ of bad words, which are not allowed to be part of the words we are looking for. We derive a generating function $f(s)$ with the coefficient of $s^n$ being  the number of searched words of length $n$.

According to the paper (p.7) the generating function $f(s)$  is
\begin{align*}
f(s)=\frac{1}{1-ds-\text{weight}(\mathcal{C})}\tag{1}
\end{align*}
with $d=|\mathcal{V}|=2$, the size of the alphabet and $\mathcal{C}$ the weight-numerator of bad words with
\begin{align*}
\text{weight}(\mathcal{C})=\text{weight}(\mathcal{C}[10100101])
\end{align*}

We calculate according to the paper
  \begin{align*}
\text{weight}(\mathcal{C}[10100101])&=-s^8-s^5\text{weight}(\mathcal{C}[10100101])\\
&\qquad\quad-s^7\text{weight}(\mathcal{C}[10100101])\\
\end{align*}
and get
  \begin{align*}
\text{weight}(\mathcal{C})=-\frac{s^8}{1+s^5+s^7}
\end{align*}
It    follows
\begin{align*}
f(s)&=\frac{1}{1-ds-\text{weight}(\mathcal{C})}\\
&=\frac{1}{1-2s+\frac{s^8}{1+s^5+s^7}}\\
&=\frac{1+s^5+s^7}{1-2s+s^5-2s^6+s^7-s^8}\\
\end{align*}

The coefficient $[s^n]$ in $f(s)$ gives the number of binary  words of length $n$ which does  not contain  $10100101$. Since we want to count the number of words of length $20$ which do contain the word we take the generating function
\begin{align*}
\frac{1}{1-2s}=1+2s+4s^2+8s^3+\cdots
\end{align*}
which counts all binary words and subtract $f(s)$ from it.

We obtain with some help of Wolfram Alpha
  \begin{align*}
\frac{1}{1-2s}-\frac{1+s^5+s^7}{1-2s+s^5-2s^6+s^7-s^8}&=s^8+4s^9+12s^{10}+32s^{11}+\cdots\\
&\qquad+\color{blue}{51\,806s^{20}}+\cdots
\end{align*}
  in accordance with the calculation  above.
Comment: Ah! So you have to add back the substrings that occur at least three times because they were already counted (and excluded) from the case in which it occurs *at least* two times and thus three times or more as well? Thank you!
Comment: @Fehniix: Correct. You're welcome. :-)
Comment: And also, were it the case in which there was a substring repeating itself four times, I would have to subtract the strings that repeat themselves at least two times, add back those that repeat at least three times, and since those that repeat themselves four times are already included in those that repeat at least three times, I would have to exclude them, correct?

Bitstrings and set of solutions


Identify groups of identical rows in a matrix

Question: tl;dr What is the idiomatic way to identify groups of identical rows in a matrix in R?

Given an n-by-2 matrix where some rows occur more than once,
> mat <- matrix(c(2,5,5,3,4,6,2,5,4,6,4,6), ncol=2, byrow=T)
> mat
     [,1] [,2]
[1,]    2    5
[2,]    5    3
[3,]    4    6
[4,]    2    5
[5,]    4    6
[6,]    4    6

I am looking to get the groups of row indices of identical rows. In the example above, rows (1,4) are identical, and so are rows (3,5,6). Finally, there is row (2). I am looking to get these groups, represented in whatever way is idiomatic in R.
The output could be something like this,
> groups <- matrix(c(1,1, 2,2, 3,3, 4,1, 5,3, 6,3), ncol=2, byrow=T)
> groups
     [,1] [,2]
[1,]    1    1
[2,]    2    2
[3,]    3    3
[4,]    4    1
[5,]    5    3
[6,]    6    3

where the first column contains the row indices of mat and the second the group index for each row index.  Or it could be like this:
> split(groups[,1], groups[,2])
$`1`
[1] 1 4

$`2`
[1] 2

$`3`
[1] 3 5 6

Either will do. I am not sure what is the best way to represent groups in R, and advice on this is also welcome.

For benchmarking purposes, here's a larger dataset:
set.seed(123)
n <- 10000000
mat <- matrix(sample.int(10, 2*n, replace = T), ncol=2)

Answer: cbind with sequence of rows and the match between the rows and unique values of the row
v1 <- paste(mat[,1], mat[,2])
# or if there are more columns
#v1 <-  do.call(paste, as.data.frame(mat))
out <- cbind(seq_len(nrow(mat)), match(v1, unique(v1)))

-output
> out
     [,1] [,2]
[1,]    1    1
[2,]    2    2
[3,]    3    3
[4,]    4    1
[5,]    5    3
[6,]    6    3

If we want a list output
split(out[,1], out[,2])

-ouptut
$`1`
[1] 1 4

$`2`
[1] 2

$`3`
[1] 3 5 6

Benchmarks
With the OP's big data
> system.time({
+ v1 <- paste(mat[,1], mat[,2])
+
+ out <- cbind(seq_len(nrow(mat)), match(v1, unique(v1)))
+
+ })
   user  system elapsed
  2.603   0.130   2.706

Comment: @Szabolcs ```library(rbenchmark)``` is a nice and simple thing
Comment: Thanks! I am new to R. Is it considered good practice to convert things to strings, as in `paste` here? Is this going to be detrimental to performance?
Comment: `paste` and `match` are vectorized.  Can you check the timings with microbenchmark on a large data
Comment: Thanks for the update. I'm happy with the timings I get, I just wanted to make sure that doing such things (convert numbers to strings) is considered an acceptable thing in R, and not a hack. I'm new to R, so I do not know what is idiomatic.
Comment: @Szabolcs with `matrix`, it is a vector with dim attributes.  So, if we directly do the `match`, it will have the issue of not keeping the dim attributes


Why does G-Major work well within a C-Minor progression?

Question: I have been playing and composing Piano for many years, but never actually learnt any music theory!
So, recently, I started reading up and comparing with things I've written to understand the theory behind the music.
There is this one progression i just don't understand why it works so well...
The part i understand:
C-Minor -> F-Minor -> Bb-Major -> Eb-Major -> C-Minor
Easy... All very close in the circle-of-fifths
But then...
C-Minor -> F-Minor -> Bb-Major -> G-Major -> C-Minor ... G-Major seems a completely unrelated chord in the circle of fifths, and yet works great (to transition to a higher inversion in an uplifting way).
Can someone please explain the theory why this G-Major works so well?
Also, I was wondering whether I can use this phenomenon to modulate to another progression close to G-Major (E-Minor?)? And if so, how? (and why?)
Thanks in advance for your answer and your patience, and apologies if my terminology is wrong/simplistic... I'm really a beginner to the theory! :-)
Answer: The G major chord is the "usual" dominant in the key of C minor. The harmonic point is the half-step approach to the note C from B. It is extremely common.
As a side note, often, when a phrase is repeated like Cm-Fm-Bb-Gm-Cm-Cm-Fm-Bb-G7-Cm would be common. The opening is a section of the cycle of fifths (or "circle" or "fourths" depending on the author) and the first Gm-Cm doesn't have a strong cadential (or ending) feel but the G7-Cm (or G-Cm) does.
Comment: It's better to learn how minor key harmony works and not think of it terms of three different minor scales, replacing the dominant chord, etc. Simply put a G major chord is not unrelated or foreign to the key C minor. It's absolutely essential in defining the key.
Comment: When playing a G to Cm progression, the  B in the G chord moves by half-step up to C. In a Gm to Cm progression, the Bb also moves to C but by a whole-step; there is a difference in sound and composers have tended to use the half-step approach at closures. Neither is wrong, just different. In Medieval music, one two-voice closure would have been a major sixth (D-B for example) expanding to an octave (C-C); using G rather than Gm contains this pattern. In a practical sense, one can use the Gm to Cm progression inside a phrase and the G to Cm progression at cadences to alert the listeners.
Comment: I guess you say "usual" because the dominant in a natural minor scale is a minor chord, but that it is extremely common to replace it with a major (or major seventh). correct?
"The harmonic point is the half-step approach to the note C from B" - could you or someone else please explain what this means ?
Comment: @MichaelCurtis Any advice on how i can "learn how minor key harmony works"? Things i can do/learn by myself... (I had started private lessons, but had to stop after just 2 lessons nearly a year ago due to Covid...)
Comment: @Stevo, that's got to be disappointing. IMO you should study with "a book and Bach!" I just coined that phrase :-) Take a look at my answer for some details. It's a bit long for a comment.
Answer: G major contains G B and D. G and D are already diatonic to the scale of C minor. In other words, they are part of that scale.
The plot thickens, though, when we find out that there are different minor scales. The first is a direct steal from its relative major. So, in key Cm, all the notes come from the E♭ major scale. Which contains B♭.
Then there's two others - the harmonic and melodic minor scale sets.Both of which ditch the B♭ note in favour of B♮. And that's where the magic happens. We can use G B D (as in Gmajor), within key C minor. That B note is the leading note, just one semitone under the tonic, here, C. That gives a sonic pull towards the C, and is the main reason why G major sounds so good in key C minor.
2nd question - not quite so easy, as when we're on that G, it sounds like (and is) leading somewhere - the C. So just getting to G and thinking we're there doesn't work. Try it and listen.
Answer:
...All very close in the circle-of-fifths...completely unrelated chord in the circle of fifths...

The circle of fifths isn't the "be all end all" of harmony. You should also learn about counterpoint and voice leading. But, let's stick with the circle of fifths for now, but with one change: rather than a literal circle of fifths consider root progression by descending perfect fifth.

C-Minor -> F-Minor -> Bb-Major -> Eb-Major -> C-Minor ... All very close in the circle-of-fifths

The root progressions are ↓P5 ↓P5 ↓P5 ↓m3, and the descending third motion connections relative major/minor chords.

C-Minor -> F-Minor -> Bb-Major -> G-Major -> C-Minor ... seems a completely unrelated chord in the circle of fifths

The root progressions are ↓P5 ↓P5 ↓m3 ↓P5, again the descending third motion connections relative major/minor chords.
Both progressions give the tones for a key signature of three flats with C minor being the sensible tonic. The progression G-Major -> C-Minor nominally gives us a cadence. Strictly speaking a cadence would depend on voice leading and rhythm, but the progression is tonally very important. Such cadential harmony would be one of the very first topics covered in a harmony textbook.
Identifying the cadential harmony is the basic answer to your question, but I think you should also consider your basic premise about the harmony that one of the progressions is more related to the circle of fifths. Both have the same number and type of root progressions. Both use chords from C minor. The only difference relative to the circle of fifths and roots by descending fifths is the placement of the descending third progression. The two progressions aren't very different on that basis.
I'm guessing you regard the G major chord as unrelated, because you are expecting it to be a G minor chord. That would be the case if you used only the purely diatonic chords from a key signature of three flats. But that is not how minor key harmony works. In the key of C minor the basic circle of fifths would be: Cm Fm Bb Eb Ab Ddim G Cm. The G chord is chromatic in terms of the key signature, but it does that by convention in C minor. The seventh degree in minor keys is always raised to one half step below the tonic for dominant chord. Some say this minor key harmony uses chord derived from the harmonic minor scale, but that just makes a different muddle of minor key harmony. Suffice to say minor key harmony is considered a more advanced harmony topic usually covered in the later sections of a textbook or course.
Your first progression is a plain vanilla diatonic progression that returns to the tonic with a relative major/minor progression.
Your second progression "works well" because you inadvertently put it into conventional minor key harmony with the standard dominant chord and a descending fifth, cadential progression back to the tonic.

Also, I was wondering whether I can use this phenomenon to modulate to another progression close to G-Major (E-Minor?)? And if so, how? (and why?)

Yes. Basically harmony is about relative relationships between chords. They typical thing is label chords with Roman numerals to analyze and list progressions. Your two would be:

Cm: i iv VII III i
Cm: i iv VII V i

Just transpose them to another minor key:

Em: i iv VII V i
Em Am D B Em

Nothing about the progression changes in the transposition except the tonic.
I'm not sure if you really meant to take your minor progressions and put them into a major key (G major.) That won't always be a mechanical process of transposition. Sometimes you can just "flip" the chord qualities of major, minor, diminished from those of minor keys to those of major keys:

Major: i iv VII  III VI iio V i
Minor: I IV viio iii vi ii  V I

...where the o means "diminished." So, in major:

Em: i iv VII  V i
G:  I IV viio V I
G C F#o D G

...which should work just fine. Other progressions might need tweaking to flip from minor to major and vice versa, but that would involve a deeper study of harmony.

Some harmony resources:
JS Bach

Two-Part Inventions
371 Harmonized Chorales
Anna Magdalena and Wilhelm Friedemann notebooks.

The music of Handel would also be a great resource. Historically Baroque composers will provide for minor key examples than classical composers like Mozart. This style may not be your style, but it would be the stuff to study to learn common practice for minor key harmony.
Harmony Textbooks

first read up on the rule of the octave, it will provide a concise rule of thumb for practical harmony and was part of how harmony was taught historically
Kostka, Tonal Harmony is a well known textbook, you would need to purchase it
Some older textbooks can be found in Google Books (use the preview and free ebook filters) or the Internet Archive. They aren't always great - teaching and writing approaches change over time - but they are free. This one - Hamilton, A Manual of Harmony - seem pretty good, it gives a lot of major/minor scale harmonization examples side by side and has a section on cadences.

I think it's best to compare several textbooks and then reconcile what they say in regard to real historic scores. Ultimately what great composers did is what matters most. Textbooks attempt to describe that with different wordings, some better than others, sometimes contradictory. But you can always rely on real composer's scores.
Answer: A V in a minor progression, especially before the i, sounds pleasant because the third of the V is a one semitone below the root and acts as a leading tone which resolves up to the tonic.
Examples: The chorus of Eastbound and Down (Em C A B7) [as a bonus, the fifth of B7 is also a leading tone to the G major that starts the verse in E&D] and the verse of some versions of Poor Wayfaring Stranger (Em Am Em Am B7 Am).
I can't think of any good examples immediately where the fifth is a triad and not a dominant 7, but I have heard it. If anyone comments, I'll add.
Comment: E.g. in lydian mode chord on the 5th step is maj7
Comment: `i v6 iv6 V` is a common example where the dominant is not always a dominant seventh chord or major triad.
Answer: If we  play in Cm and want to have a leading tone we have to augment the 7th tone Bb => B. (This is the historical evolution of the music!) Now, this transformation we have a major dominant chord G,B,D which is borrowed from C-major, the parallel of Cm.
After a modulation from Cm to G you can easily modulate from G to Em (relative key of G.)
E.g. :

Cm-Fm-G7-Cm = i-iv-V7-i (c-minor)
Cm-Ab-D7-G  = i-VI (V)/V (cm => G)
G-Am-B7-Em  = I-ii-(V7)/vi (G => em)
Em-Am-B7-Em = i-iv-V7-i (em)
Comment: It's not necessarily borrowed (from C major). It doesn't need to be. It already exists within the C mnor key and scales.
Answer: One analogy of composition is that it's like embarking on a musical journey. You have mentioned that both sequences feature the circle of fifths, but you haven't referenced the fact that to go through the circle of fifths is like setting out on a journey.
The first few chords in a circle of fifths is like setting out for a walk on a very well-known terrain. Then the further you go, the less well-known it gets (but the act of walking still feels familiar). In both sequences you wrote out above, you come home before completing that full walk. The act of coming home always gets you to your front door (G major, the one before you walk back into your C min house).
Those walks are a bit boring, they are over quickly (4 bars) and they repeat over and over. They feel nice because they are familiar. Only difference between your first and second sequence is that you made a point of highlighting your front door. In the first one you just walked, then presented as being back home again without mentioning the front door, i.e. no cadence as part of the sequence. This allowed you to show more of your walk within 4 bars. In both cases, they repeat ad nauseam, like a daily walk to the shops where each walk lasts 4 bars.
In music terms, you are doing what needs to get done. You are walking for functional reasons, and in pop this usually means providing a simple harmonic basis for lyrics or musical effects. (I try not to sound snide!)
When you understand that a cadence (as you described - G maj to C min) is merely the last chord in your journey, i.e. it's being back at your front door, you realise why it feels "nice".
This regular short walk may be useful and good for your health, but if you are a true explorer of harmony, if you walk for walking's sake - then you need to develop, surprise, and amaze your listener with different harmonic directions AND harmonic forces that pull you in directions.
Otherwise, there's nothing wrong with using this 4 bar walk as a means to do other things like explore lyrics or rhythm.
Pop songs go against the idea of expansion and longer term progression, which is why people get posh about classical and even some types of jazz; they are for the "real" explorer. This is often true, although I personally believe the greatest pop composers understand this and at least try and get people out for a proper walk.
Comment: I fully agree with you, even if i felt a bit offended at first with you thinking those progressions were my whole compositions :-)... But then... if I am honest, I DO admit finding my pieces too simple. This is in fact the main reason why I decided to try to teach myself more theory. Do you have any advice on how I can learn to walk further and more elaborately? Things i can do/learn by myself... I had started private lessons, but had to stop after just 2 lessons nearly a year ago due to Covid... Maybe i should ask this as a separate stackexchange question?
Comment: Wasn't my intention and I didn't think that your compositions were that simple. I'll be absolutely honest, I'm a music grad but a not a composer. I had a place to study composition at Guildhall (London) 22 years ago but didn't take it, probably best I didn't. My advice though is simple. Find some well known classical music, or maybe a Beatles tune you don't know, write down the chords or harmonic progression. Pause, then guess where the chords might go next, be amazed by how your expectations were thwarted, etc. Basically study some good quality existing music is my best advice.
Answer:
G-Major seems a completely unrelated chord in the circle of fifths

If that's so, then how is this possible: https://en.wikipedia.org/wiki/Circle_of_fifths#Jazz_and_popular_music
Fly Me to the Moon is presented as an example of the circle of fifths. Here are the beginning chords in C minor:
Cm Fm Bb Eb Ab Ddim G7 Cm.
There's a G major chord in there, in C minor. In a textbook example of the circle of fifths.
The confusion must be caused by someone having presented the circle of fifths as some kind of a "true" underlying model of music theory where everything and anything can be mapped. And I think you're even talking about a narrower related concept, the diatonic circle of fifths. https://en.wikipedia.org/wiki/Circle_of_fifths#Diatonic_circle_of_fifths
Diatonic chords and notes aren't enough for everything. If they were, you wouldn't have accidentals in music. And you could play all songs without the black keys. Every now and then, the set of intervals around the tonic is modified a little bit, for example to get the G major chord. No, it's not diatonic in C minor, but it sounds nice to get that B - C pitch movement. The B note is soooo close to being home, it makes you want to get back home to C much more than a Bb would. Tonic Sweet Tonic.
Comment: Depending on how we perceive diatonic - B's in there, in key C minor.
Comment: @Tim there's no B in C minor. There is in the harmonic and melodic *scales*, but diatonic by definition refers to the natural notes of a given tonality.
Comment: @Tim If we include A and B in C minor as "diatonic", then I'll have to start calling the actual diatonic scales something like The Scales Formerly Know As Diatonic. Raising Bb to B is just a very, very commonly used harmonic alteration in songs in the key of C minor. It's more common to use it than not. If you want to have a proper dominant chord, you have to make that change for the duration of the chord. Some people would like to have diatonic mean "anything that's commonly used". Laurence talks about b7 being "honorary diatonic". It's just a very commonly used alteration, not diatonic.
Comment: Indeed, my confusion comes from the (wrong) understanding that proximity in the circle of fifths defines harmony. If my (still limited compared to you guys) understanding is correct, proximity is related to diatonic chords. And yes, I used the G-Major to go back to the Tonic, moving to a higher inversion, which gives a very 'uplifting relief' feeling.
Comment: @Stevo If you can locate the chord tones and scale degrees relative to the home note at each point, and if you can see how the sounding voices move or change, then you understand harmony. I get a feeling that you already understand harmony in a practical sense. There's no magic formula, or if there is, the circle of fifths isn't it. You can see on the piano keyboard if the bass moves by a fifth/fourth, and if you keep making those steps, eventually you'll end up where you started. So "circle". It's really simple, I don't get what the big deal is. Forget the circle hoo-ha if it bothers you.
Answer: C minor / F minor / G major = C harmonic minor.


Proof: If the dual map $f^*$ is the null map, then $f$ is the null map

Question: Let $V$ and $W$ be vector spaces, and let $f:V\to W$ be a linear map. Let $f^*:W^*\to V^*$ be the dual map ($f^*=f^T= f$ transposed). Prove that if $f^*$ is the null map, then $f$ is the null map.
Comment: Hello, welcome to Math.SE. Thank you for your question! It is good practice on this site to add a bit of information on the context your question came up in, and to share your own work on it. It's also fine if you state that you're completely lost -- the information is helpful for answerers to gauge their answer on. For more information on asking a good question on this site, see [here](http://meta.math.stackexchange.com/q/9959/43351).
Answer: If $f^\ast$ is the null map, then for any $\varphi:W\to K$ we have that $\varphi f:V\to K$ is the null map. This means that ${\rm im \;}f\subseteq\ker \varphi$ for every $\varphi\in W^\ast$. What is $$\bigcap_{\varphi\in  W^\ast}\ker\varphi\;?$$


Subversion Edge LDAP (require CAC Certificate not Username and Password)

Question: What I've Done:
I've successfully installed and configured Subversion Edge 3.1.2 with LDAP support on a Windows 2008 server. I have configured LDAP users and am able to use LDAP credentials to work on repositories just fine. No issues whatsoever. Works great!
What I Want To Do:
I've been searching for several hours now in hopes to find some information on how to configure Subversion Edge server to require client certificates for user authentication against an LDAP environment. I have not found anything yet that gives me an indication of how to do it. I know there are SVN clients that are capable of prompting for CAC certificates but I cannot figure out how to set my server up to require it.
NOTE: CAC authentication is already setup and working in the windows environment. I have TortoiseSVN client which has CAC support enabled.
Desired Outcome:
When running svn commands that require authentication against my Subversion Edge Server I want it to prompt me for my CAC certificate instead of my Active Directory username and password.
If anyone has any information on this I'd greatly appreciate it.
EDIT: I'm still digging so if I find out anything I'll update this question with what I found. I've made some progress and I believe I've been searching for the wrong information. This apparently is an Apache configuration issue. I added SSLVerifyClient require which I saw in sample Apache configuration to my httpd.conf and now my SVN client prompts me for a certificate file rather than a username and password.
UPDATE 14 SEP 2012
Okay I've got it working somewhat. This isn't a subversion issue as much as it is an Apache configuration issue. I've gone down the road of using the SSPI module and I've got it mostly working the way I want. There are still some issues with Internet Explorer viewing the SSL protected sites for browsing the svn repositories. It will prompt for my CAC certificate just fine and then after that I get a page that says Internet Explorer can't view the website. I've found some information online that may address this but haven't got it to work just yet. Also, I've having some issues now with AnkhSVN Visual Studio plugin not caching the client certificate credentials. I'm being prompted far too many times when I do various SVN related things.
RESOURCES:
http://www.eperezdesigns.com/blog/articles/enable-dod-cac-authentication-on-apache/
Getting Apache configured for DOD CAC (this is mostly working for me, still have Internet Explorer issues with the SSL configuration apparently, however, works fine in Firefox).
http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-serversetup-apache.html
This has various configuration information for configuring Apache so that you can get it to talk to a Windows domain.
Comment: If you find the answer on your own, be sure to post it as the answer to your question and accept it, to be more useful to any future visitors.
Comment: I haven't found a suitable fully working solution to this yet. I've given up for now and am using username/password.
Answer: If you are having problems using a CAC to access a site with IE, but the site works with the CAC in Firefox, you might be seeing a Microsoft cross-certificate chaining issue.
From DISA's guidance in the FBCA Cross-Certificate Remover Tool User Guide:

The Federal Bridge Certification Authority (FBCA) Cross-Certificate
  Remover Tool is designed to help DoD organizations address the
  Microsoft cross-certificate chaining issue. The issue may manifest
  itself in several ways:

Users may be unable to access DoD web sites normally accessible using certificates on their Common Access Cards (CACs)
DoD signed emails in Outlook may appear invalid
Users may experience extensive delays with Outlook or Internet Explorer during validation
Users' CAC certificates may appear to chain to a root beyond/other than DoD Root 2
Users may receive a prompt to install the Common Policy Root Certification Authority (CA) or other roots cross-certified with the
  Federal Bridge when opening a signed email from a DoD sender whose
  workstation is misconfigured

Run the FBCA Cross-Certificate Remover Tool to address the error. The user's guide describes some additional steps you might have to take.
Comment: Thanks for responding. CAC configuration was abandoned and username/password auth was used.


Extraindo as palavras de um texto longo e criando estatísticas das mesmas. O que está errado?

Question: Temos o livro "Pride and Prejudice by Jane Austen" do projeto Gutenberg:
http://www.gutenberg.org/ebooks/1342
O objetivo é extrair todas as palavras do texto, criando estatísticas como:
frequência de cada palavra, total de caracteres no texto, tamanho médio da palavra, tamanho médio de frase e um "top 10" das palavras mais longas.
Observando o texto, verifiquei que muitas palavras contêm caracteres estranhos como:
"the,"

Exigindo que se removam inicialmente estes caracteres.
Como eu tentei fazer:
# -*- coding: UTF-8 -*-
from string import punctuation
from collections import Counter
with open("1342-0.txt",encoding='utf8') as f:
    texto = f.read()

words = texto.split()
n_words =[]
for word in words:
    for p in punctuation:
        if p in word:
            n_word = word.replace(p,"")
            n_words.append(n_word)
        n_words.append(word)

"1342-0.txt" é o livro em questão.
O código acima tenta eliminar os caracteres indesejados mas não funciona. O que está errado?
Alguma ideia melhor?
Comment: Não entendi o que tem de estranho em `"the,"`. É a virgula? Se for é só remover toda pontuação antes de processar o texto.
Comment: @Augusto Vasques: isso q estou tentando! Remover pontuacao, acentos... mas nao funcionou –
Answer: Uma ideia inicial é fazer o split não só por espaços, mas por qualquer caractere que não faça parte de uma palavra:
from collections import Counter
import re

r = re.compile(r'\W+')
c = Counter()
with open("1342-0.txt", encoding='utf8') as f:
    for linha in f:
        for word in r.split(linha):
            c.update([word])

print(c)

O atalho \W é "tudo que não for letra, número, ou o caractere _"  - e como no texto tem "palavras" como _she_, isso é considerado uma palavra diferente de she. Eu também considero que os números (como 1) são "palavras", que também são contabilizadas.
Conforme encontro as palavras, vou atualizando o Counter usando método update (se a chave não existir, ela é criada com o valor 1, e caso ela exista, soma-se 1 ao seu valor - no final temos a contagem total de cada palavra).
Outro detalhe é que read() carrega todo o conteúdo do arquivo para a memória, de uma vez. Dependendo do tamanho do arquivo, isso pode ser um problema. Já o código acima lê uma linha por vez (e estou assumindo que não existe um caso de uma palavra começar em uma linha e terminar em outra - se bem que neste caso, usar read e split também não iria considerar que é a mesma palavra).
Se você não quiser incluir o _ como parte de uma palavra, basta mudar a regex para:
r = re.compile(r'[\W_]+')

O problema é que também há palavras com hífen, como over-scrupulous. O código acima considera que são duas palavras diferentes ("over" e "scrupulous"). Se quer que elas sejam uma palavra só, tem que mudar um pouco:
from collections import Counter
import re

r = re.compile(r'\b\w+(?:-\w+)*\b')
c = Counter()
with open("1342-0.txt", encoding='utf8') as f:
    for linha in f:
        for word in r.findall(linha):
            c.update([word])

print(c)

Agora eu uso \w+ (um ou mais caracteres que formam uma palavra), e coloco um trecho contendo hífen e \w+ (e todo este trecho pode se repetir zero ou mais vezes). Assim pego palavras com um ou mais hífens também.
Se não quiser incluir o _ como parte de uma palavra, use:
r = re.compile(r'\b[^\W_]+(?:-[^\W_]+)*\b')

Vale lembrar que string.punctuation só considera os caracteres !"#$%&'()*+,-./:;<=>?@[]^_{|}`. Se tiver qualquer outro caractere no texto que não seja letra, número ou _, ele não será removido.
Um exemplo é o caractere " (presente no texto), que não é a mesma coisa que " (são aspas diferentes, o primeiro é o "LEFT DOUBLE QUOTATION MARK" e o segundo é "QUOTATION MARK", e se você usar punctuation, só removerá o segundo).
Comment: Obrigado! GOstaria de entender o que estou fazendo de errado na hora de tirar os caracteres indesejados das palavras. Alguma ideia?
Comment: @LaurindaSouza Não entendi bem sua lógica, mas parece ser porque vc faz `n_word = word.replace(p,"")` (ou seja, a palavra sem a pontuação fica em `n_word`, mas no final vc faz `n_words.append(word)` (ou seja, adiciona a palavra original, com as pontuações). Ficou um algoritmo meio confuso, e acho até que tem `append` a mais, uma versão mais simples do seu loop seria: https://ideone.com/LnJf3v
Comment: se puder ajudar   https://pt.stackoverflow.com/questions/443768/utilizando-recursos-de-programa%c3%a3o-funcional-para-remover-uma-lista-de-palavras-d
Answer: Para remover os caracteres de pontuação de um texto em python basta só uma linha:
from string import punctuation

texto = '''It is a truth universally acknowledged, that a single man in
      possession of a good fortune, must be in want of a wife.

      However little known the feelings or views of such a man may be
      on his first entering a neighbourhood, this truth is so well
      fixed in the minds of the surrounding families, that he is
      considered the rightful property of some one or other of their
      daughters.
'''

#Remove os pontuadores
print(texto.translate(str.maketrans('', '', punctuation)))

Resultando:
  It is a truth universally acknowledged that a single man in
  possession of a good fortune must be in want of a wife

  However little known the feelings or views of such a man may be
  on his first entering a neighbourhood this truth is so well
  fixed in the minds of the surrounding families that he is
  considered the rightful property of some one or other of their
  daughters

Funcionando no Repl.it: https://repl.it/repls/DarkcyanPointlessChord
Exemplo 2: https://repl.it/repls/ResponsibleVariableTheories
A lógica é a seguinte, o método str.translate() retorna uma cópia da string na qual cada caractere foi mapeado através da tabela de conversão especificada pelo método str.maketrans().
Comment: Não conhecia este translate!
Comment: Não queria deixar como resposta porque quase não dá conteúdo de tão simples. Eu ia deixar como comentário na sua pergunta.
Comment: @LaurindaSouza O problema é que `punctuation` só considera alguns caracteres ASCII, mas no texto tem por exemplo o caractere `"`, que **não** é a mesma coisa que `"` (o primeiro não é removido se usar `punctuation`, somente o segundo é).
Comment: @hkotsubo é só adicionar o caractere `"` na tabela tradução. Não tem mistério.
Comment: Sim, Augusto, é que aí ela vai ter que saber todos os caracteres não-ASCII que tem no texto e incluí-los na lista. Dependendo do texto, pode ser mais trabalhoso...
Comment: @Augusto Vasques: terei que utilizar str.maketrans('', '', punctuation) para cada caracter? Não entendi bem... Aqui só está removendo as vírgulas...
Comment: @LaurindaSouza ele tirou tudo que está definido em [`string.punctuation`](https://docs.python.org/3.8/library/string.html#string.punctuation). O que estamos discutindo é sobre o caractere `"` que não está definido em `punctuactor` e eu disse que é só concatenar o caractere `"` a `string.punctuation` para que seja também removido.\
Comment: @Augusto Vasques: entendi! Sou iniciante na Programação! Obrigado!
Comment: @LaurindaSouza olha esse outro exemplo https://repl.it/repls/ResponsibleVariableTheories que remove os caracteres `"` e `"` .É sobre isso que estávamos discutindo, que caso queira remover algo que não esteja definido em  `string.punctuation` faz-se necessário adicionar e eu defendo que a adição é simples.
Comment: @LaurindaSouza A adição é simples se vc sabe quais caracteres estão presentes no texto. Mas se surgir um texto novo com um caractere não previsto, terá que mudar o código e adicioná-lo também. Já a minha resposta não tem esse problema, já elimina tudo que não é parte de uma palavra - tudo bem que tem outros problemas, como o fato de usar regex, que não é lá tão simples e nem tão performático :-)
Comment: @Augusto Vasques Por que chamar str.maketrans e nao texto.maketrans ?
Comment: @LaurindaSouza Leia na documentação que [str.maketrans()](https://docs.python.org/3/library/stdtypes.html#str.maketrans) é um método estático e métodos estáticos devem ser chamados pela classe e não por instâncias.
Comment: @LaurindaSouza Já o [str.translate()](https://docs.python.org/3/library/stdtypes.html#str.translate) é um método de instancia o que significa que deverá ser evocado por uma instância `texto.translate(...`.


Test if a Popen process is waiting for input

Question: I start a process with POpen and under normal circuimstances it should just do a job and write things to stdout which I then capture.  In exceptional cases the process will fallback to an interactive mode and wait for user input.  How can I detect that case and react appropriately?
Comment: Read its stdout and check for a prompt?
Answer: You can always send something to sub-process even it read nothing. So you just sent to sub-process, if it works OK, then those staff you sent will be dropped, if it failed, then you will read response.


Unrolling list into function args?

Question: Given an argument list and a two-arity function:

args = [5, 6]
f = ((y,z)->y*z)

How do I unroll args into function arguments? - E.g. in Python you can do: f(*args).
What I've tried (more JavaScript style I suppose):

Function.call.apply(f, args)
# When that failed, tried:
((y,z) -> y*z).call.apply(null, [5,6])

Answer: Use f(args...).
f = (x, y) -> x + y
list = [1, 2]
console.log(f(list...)) # -> 3

You can also mix and match this with regular arguments:
f = (a, b, c, d) -> a*b + c*d
list = [2, 3]
console.log(f(1, list..., 4)) # -> 1*2 + 3*4 == 14

Unrolling list into function args?


OLS $\hat{\beta}$ definition

Question: Why is $\hat{\beta}$ equal to $$\left(\sum_{i=1}^n x_i x_i' \right)^{-1} \sum_{i=1}^n x_i y_1  = \left(\textbf{ X}^{'}\textbf{X} \right)^{-1} \textbf X'\textbf y \ ? $$ I am rather confused.
Comment: This question needs some serious context. Please provide more details and background. Where does this equation come from?
Comment: Pls see here https://math.stackexchange.com/questions/1836779/derivative-of-dot-product-of-residual-sum-square-in-matrix-notation/1836790#1836790
Answer: Firstly note that $\textbf X $ is  a $n \times (k+1)$ matrix, where $k$ represents the number of predictor variables.
The columns of the $X$ matrix are
$x_j=\begin{pmatrix}{}  x_{ 1j}  \\   x_{ 2j}  \\ \vdots \\  x_{ nj}  \end{pmatrix} \quad \forall \ \ 0\leq j\leq k$
where $x_0=\begin{pmatrix}{}  1  \\   1  \\ \vdots \\  1 \end{pmatrix}$
That means that X is equal to
$\textbf X=\begin{pmatrix}{} 1 & x_{ 11} & x_{ 12} &\ldots & x_{ 1k}\\  1& x_{ 21}  & x_{ 22} &\ldots & x_{ 2k}\\ 1& \vdots & \vdots & \ddots & \vdots \\ 1& x_{ n1} & x_{ n2} &\ldots & x_{ nk} \end{pmatrix} $
where $1\leq i \leq n$
Therefore you have to show, for instance, that for $n=2$ and $k=2$
$$\sum_{i=1}^2 x_i\cdot x_i^T=\sum_{i=1}^2 \begin{pmatrix}{} 1 \\ x_{ i1} \\ x_{ i2} \\  \end{pmatrix}\cdot \begin{pmatrix}{} 1 & x_{ i1} & x_{ i2} \end{pmatrix}\qquad (\color{blue}I)$$ is equal to $$
\begin{pmatrix}{} 1 & 1 &  \\  x_{ 11}&  x_{ 21}  \\ x_{ 12}& x_{ 22}  \end{pmatrix}\cdot \begin{pmatrix}{} 1 & x_{ 11} & x_{ 12} \\  1& x_{ 21}  & x_{ 22}    \end{pmatrix} \qquad (\color{blue}{II})$$
Hint:
For the summand $i=1$ you should get at $(\color{blue}I)$
$\begin{pmatrix}{} 1 & x_{ 11} & x_{ 12} \\  x_{ 11}& x_{ 21}^2 & x_{ 11}\cdot x_{ 12}\\ x_{ 12}& x_{ 12}\cdot x_{ 11}& x_{ 12}^2\end{pmatrix} $
$\small{\text{I assume your equality appears in context of ordinary least squares.  }}$
Answer: Taking a different approach to calculus' one can proceed directly from your orthogonality assumptions.
By hypothesis (i.e. our exogeneity assumption), $\mathbb{E}[X'u] = 0$.  But
$$
u = y- X\beta
$$
hence:
$$
\mathbb{E}[X'u] = \mathbb{E}[X'y - X'X\beta] = 0
$$
or, re-arranging:
$$
\mathbb{E}[X'y] = \mathbb{E}[X'X]\beta
$$
and hence our 'true' $\beta$ satisfies:
$$
\beta = \mathbb{E}[X'X]^{-1}\mathbb{E}[X'y].
$$
Now by what's called the 'analogy principle' (which is a nice way of saying 'familiarity with the weak law of large numbers/CMT') we can 'operationalize' this definition by the following estimator:
$$
\hat{\beta} = \bigg[\sum_i^NX_i' X_i\bigg]^{-1}\sum_i^N X_i' y_i
$$
Answer: As @caradonna states, by Pythagoras, the error vector "$y - X'\hat{\beta}$" must be orthogonal to the subspace spanned by the covariates, $\text{range}(X)$, since otherwise we could improve our fit by adjusting $\beta$ to take advantage of this unused correlation.
This implies that the least squares fit has the following property: $$
X' (y - X\hat{\beta}) = 0,$$ which is equivalent to saying that $$X'y = X'X\hat{\beta}$$ or alternatively that $\hat{\beta} = (X'X)^{-1}X'y$
This of course assumes that $(X'X)^{-1}$ is invertible. If this is not the case, then there exists an infinite number of $\hat{\beta}$ which minimize the squared error. People might then use the pseudo-inverse $(X'X)^+$ to pick the least-norm solution. I.e., pick the $\hat{\beta}$ for which $\|\hat{\beta}\|$ is smallest.
What's interesting here is that the prediction vector $\hat{y} = X \, \hat{\beta} = X\,(X'X)^{-1}X'y$ turns out to be the orthogonal projection of $y$ onto the space spanned by the columns of $X$. The matrix representing this projection is $X(X'X)^{-1}X'$.

OLS $\hat{\beta}$ definition


Finding the derivative?

Question: Hello everyone how would I find derivative $f'(-2)$ if
$$f(x)=(x^3-5)(x^2-3)(x+1)$$
What I did is I made $(x^2-3)(x+1)$ the other part in the product rule.
This is what I did.
$(x^3-5)(2x^2+2x-3)+(3x^2)(x^3+x^2-3x-3)$
Then I plug I in $x=-2$ and got $-25$ but I am not sure if I did it correctly.
Comment: I think the second factor in the first term should have an $3 x^2$, not a $2 x^2$.
Comment: It should be $3x^2+2x-3$ for the second factor of your first term.
Comment: Mistake in differentiating $x^3+x^2-3x-3$.
Comment: Oh I see let me see If I get a correct answer.
Answer: Your strategy works fine here, since $(x^2 - 3)(x+1)$ is fairly easy to expand:
$$f(x) = (x^3 - 5)[(x^2-3)(x+1)]=(x^3 - 5)(x^3 + x^2 - 3x -3)$$
Using the product rule:
$$f'(x) = (x^3-5)(\color{blue}{\bf{3}}x^2+2x-3)+(3x^2)(x^3+x^2-3x-3)$$
(i.e., your only error was in miscalculating the derivative of $x^3$..
Now, evaluate $f'(-2)$. And if you'd like to check your work, feel free to hover the calculation below.

$$f'(-2) = -77$$

Note: The product rule for two factors can be expanded to a product rule for three factors, which is handy, to use with more complicated polynomial factors, and/or other functions of $x$: If you have a function that is a product of three functions,  e.g.
$$f(x)=g(x)\cdot h(x)\cdot j(x)$$
Then can apply use the following (Product Rule, three factors):
$$f'(x) = \left(g'(x)\cdot h(x)\cdot j(x)\right) + \left(g(x) \cdot h'(x)\cdot j(x)\right) + \left(g(x)\cdot h(x)\cdot j'(x)\right)$$
Comment: Yup...that's what you want! Your only error was in the mis-calculation  of $(x^3)\,'$
Answer: Here is a suggestion if you have a longer product. The calculation will not appear to make full sense, since we will be taking the logarithm of things that may be negative. But everything in fact works out fine.
We use $n=4$, but the method is general. suppose we want the derivative of $y$, where $y=f_1(x)f_2(x)f_3(x)f_4(x).$
Taking the logarithm, we find (?) that
$$\log y=\log f_1+\log f_2+\log f_3+\log f_4$$
Thus
$$\frac{y'}{y}=\frac{f_1'(x)}{f_1(x)}+ \frac{f_2'(x)}{f_2(x)}+\frac{f_3'(x)}{f_3(x)}+\frac{f_4'(x)}{f_4(x)}.$$
Multiply through by $y$, that is, by $f_1(x)f_2(x)f_3(x)f_4(x)$ to find $y'$.
With equal ease, the procedure handles quotients whose numerators and denominators are products.
Answer: The answer I believe should be -77, to check your work.
I'm going to expand a bit on what it is you are doing.
A completely valid way is to multiply out all the terms and take the derivatives that way.
A faster way is using the product rule.
If you have multiple products of functions:
$$f=g\cdot h\cdot j$$
Which in your case you do:
$$f(x)=(x^3−5)(x^2−3)(x+1)$$
$$g=(x^3-5)$$
$$h=(x^2-3)$$
$$j=(x+1)$$
With Derivatives
$$g'=3x^2$$
$$h'=2x$$
$$j'=1$$
You apply the following technique (Product Rule)
$$f'=g'\cdot h\cdot j + g \cdot h'\cdot j+g\cdot h\cdot j'$$

Finding the derivative?


Typescript function overloads too big files

Question: Is there a way to import/export function overloads or some other tactic for reducing the noise in the file with the actual functions? E.g. a file with around 250 lines of code that is bound to become 500+ if I keep the overloads in the same file.
Currently interface and type functions are not an option since they require casting when used for overloading purposes
Answer: Use interfaces and arrow functions
<code>// a.ts
export interface Foo {
  // overloads here...
  (a: string): string
  (): any
}

// b.ts
import {Foo} from './a'

const foo: Foo = (...args: any[]) => ...

foo()
</code>
P.S. You cannot do it with classic functions, because in that case function overloads must be placed just before the implementation.
Comment: Interfaces and types require casting(type assertion). [Example](https://stackoverflow.com/a/54554036/4132182). So, I'm using overloads with the function keyword.
Comment: @ZenVentzi Your point is not clear. What's wrong with using a variable instead of a function? in the end you use it the same way `foo()`. And no, there is no single casting or type assertion in my answer.
Comment: There is nothing "wrong" in using variable, you're missing the point. "Interfaces and types require casting(type assertion)". That's what I am trying to avoid. And again there is nothing "wrong" in that either.
Comment: @ZenVentzi I am saying again, there is *no* casting or type assertion. Type assertion is this: `123 as 123` or `{} as any`. What you are seeing is just setting a type of a variable `foo`. What's wrong with that? Why are you trying to avoid it?
Comment: Please check the code in the link above. With interfaces/types you can't achieve what the issue is there without casting. Your function is too simplistic, not even finished


Stacking without splitting data

Question: I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for training of combiner of predictions (layer-2, right figure).
In my project, I have two different multi-classification models. And I have a dataset (train/dev/test) which was used for training and testing two models.
When I have learned Stacking, I thought I tried to use the whole training set for the blending training set (layer-2), then, test the blender with the test data. Though I read the book and other websites, and they mention the training set is splitted into subsets.
Is it uncommon (or not recommended) to use the whole training set for both layer-1 and 2? I thought this is not wrong since test data has been already prepared.
I have already trained my model by the whole training dataset. So if it is not recommended, should I train my models with splitted training dataset?

The images are cited from "Hands on Machine Learning with scikit-learn and Tensorflow." (2017).
Answer: You have to have real, out-of-sample, predictions as the input to your blender, otherwise your blender is not learning about, and thereby improving, prediction accuracy - but instead learning about, and thereby improving, in-sample estimation accuracy, which can lead to overfitting.  This is why you cannot use the whole training set for both layers - if you do, some of the "predictions" made by the base models will actually be in-sample estimates, not out-of-sample predictions.
You split your training data set so that subset 1 is used to train your base models.  This is what is shown in your left picture.  Your base models then are used to generate predictions for subset 2, and these predictions, along with the actuals for subset 2, are given to your blender for training.  This is what is shown in your right picture.  Basically, the predictions are features that are given to your blender, along possibly with other features from subset 2.
The model that the blender comes up with based on subset 2 is then used to predict the test data.  This can be done by predicting the test data with each of the base models (developed on subset 1), then predicting the test data again with the blender model (developed on subset 2) + the predictions from the base models.  The resultant predictions are the ones you use for calibrating / testing the combined base models + blender model.
Alternatively, you can re-train your base models on subset 1 and 2 prior to making predictions for the test data set.  This will tend to improve the base model predictions of the test data set, but (hopefully slightly) weaken the link between the base models and the blender model, as the blender saw less-accurate predictions when it was being trained.  The blender will consequently add less value and more overfitting, but given that the base models are more accurate, it may balance out.
ETA (from comments):
In practice, I tend to split into more than 2 groups, liking 10 groups for some reason. The base models are then trained with much more data so are more accurate (at least in situations where you don't have overwhelming amounts of data) and the blender is trained on predictions from models that have accuracy characteristics that are closer to what it will see when going operational, which is a win-win in accuracy terms.
Comment: The models are not outputting predictions for the training set, they are outputting estimates for the training set.  The difference is key: predictions are out-of-sample, estimates are in-sample.  If your objective is to improve predictive power, you need to train the blender on predictions and actual values, not on in-sample estimates and actual values.  Just evaluating the blender on the test data set doesn't change the fact that it was fitted to in-sample estimates with the target data being the very same data used in the base model fitting...
Comment: Right.  In fact, I tend to split into more than 2 groups, liking 10 groups for some reason.  The base models are then trained with much more data so are more accurate (at least in situations where you don't have overwhelming amounts of data) and the blender is trained on predictions from models that have accuracy characteristics that are closer to what it will see when going operational.
Comment: So yes, it's quite likely that it will cause overfitting, unless the base models are rather incomplete and fill in each other's gaps nicely, e.g., $y = x_1 + e$, $y = x_2 + \epsilon$ as two base models would be OK.
Comment: Yes, it looks like you've got it, except that I accumulate all the 10 out of sample predictions and use them collectively, all at once, for training the blender.  This is "stacking" rather than "blending"; see https://mlwave.com/kaggle-ensembling-guide/, about 40% of the way down the page.  If you are blending, the 2-split is probably better, because the blender has more data to train on.
Comment: Well stacking doesn't train the "blender" (now "stacker") ten times, it trains it once on the collected predictions from the 10 folds.  But the base models are trained ten times for stacking and use to predict the 10 10% holdouts, so there's more work in that respect.
Comment: Thank you. I am still unconfident. How about this situation: Given two base models which were trained whole training dataset, the models output predictions for the training set.  Then, a blender learns mapping between target labels and base models' predictions. Finally, the blender is evaluated by test dataset. Does it still case overfitting?
Comment: Thank you again. Then, let me confirm my understanding is right. I need to split the training set into at least two (`train1` and `train2`, e.g., 50%-50%). The base models are trained with `train1`, the blender is trained with predictions of the base models for `train2`. Finally, `test data` is used for evaluating blender's output.
Comment: Is it like 10-fold-cross-validation? So 9 of 10 are used for training models, then the last one is used for the blender. Then, this procedure is repeated 9 times. In this theory, all training data is used for training of the blender. Though, the blender is trained with out-of-samples.
Comment: Now, it looks like I could understand the difference between blender and stacking. For example, say Blending use 10% holdout. This is the same process of 10-fold stacking, more specifically, this blending process is the one process of 10-fold stacking.  Since the stacking needs to repeat it 10 times, though blending only needs one time.
Comment: I have a final question about testing. In 10 folds setting, there are 10 base models which are trained with 10 different data. In testing stage, which base models should I use? Do I need to calculate the average over 10 base models?
Comment: In testing, you can retrain on all the train data, no holdout.  The number of base models is independent of the number of folds, BTW; I can stack a logistic regression, SVD, and random forest base model together with 10 folds, meaning there are three base models.  You'd train all three on all the train data, generate the three sets of predictions for the test data, and use the trained stacker model to combine the three sets of predictions.
Comment: OK. For example, I have three base models (LR, SVD, RandForest) in 10 folds. So there are 3*10 models (LR1,LR2,..LR10, SVD1,...) in total. In testing, test data should be fed to all 30 models. Then, the average is calculated for each LR, SVD, and RandForest. Finally, the averaged input of three models are fed to the stacker, right?
Comment: No.  You have three base models and one stacker model.  The three base models are trained 10 times each on 90% of the train data to generate a complete set of predictions that cover the entire training data.  This complete set of predictions is used to train the stacker model.  The three base models are then retrained once on 100% of the train data.  These three trained models generate predictions for the test data set, one set of predictions for each of the three models.  Those predictions are fed into the stacker model, which generates the final predictions for the test data set.
Comment: I'm sorry, still confused. I thought the three models are initialized every step (10 times) since the three models do not need to know the data for training the stacker. If I understand what you mean, the instances of the three models are just initialized at the first step, and trained 10 times. Right?
Answer: Let me explain.
In stacking you split your data in two.
One holdout set (10 - 20%).
One training set (80 - 90%).
You train your base learners individually on the training set using the same cross-validation method for each. You must use the same cross-validation fold indices for all base learners. This is because you can only train the metaclassifier on the predicted probabilities of those test-fold sections and the original raw data associated these rows; because these rows weren't used for building the base-learners.
Note: Not a single training fold can overlap a testing fold. If row 37 is inside a test fold; it cannot occur in any training fold. Otherwise you will be passing information from layer to the next. Use 10 to 20% of the training set rows as test-folds.
Now use each trained base learning to predict on the holdup set, and write new probability variables to it (Example: SVM_probs_up, LDA_probs_up).
Once the metaclassifier is trained using the test-fold probabilities from each base learner + the raw data from the same rows, then use the meta-classier to predict on the holdout set for your final results.
With Blending the training set is also split. But instead of using cross-validation you just take out 30% - 40% straight away (training-validation-set) and then train the base-learners on the remaining 60% - 70% of the training set. Then predict to training-validation-set and holdout-set, writing these as new variables to both sets. Then train the metaclassifier on the training validation set, then predict using the metaclassifier on the holdout set for your final results.

Stacking without splitting data