Clustering data into bins of variable sizes

Question: I'd like to build a model (in R or excel) that takes in large amount of linear data and segments it into "bins". The linear data is an attribute that reflects what condition that section/record is at.
I'd like to simplify the data and generalize it over sections as seen in the example below. This could perhaps be done based on a normal distribution with a confidence interval or any other method you may recommend. The idea is that to aggregate the data into more generalized bins. Each bin must have at least 4 sections/records in it but not more than 10.
So in short, I'd like to do the following:

I want a clustering method to group my data
I want each group to have 4-10 members always
I want the items in each group to be successive for example A001_002 wouldn't be grouped with A001_999

Here's an example of what I want to do followed by a data dump. Note that I have about 100,000 of these points.

Section_ID  Condition_Rating
A001_001    3
A001_002    2
A001_003    1
A001_004    3
A001_005    4
A001_006    5
A001_007    0
A001_008    0
A001_009    0
A001_010    1
A001_011    2
A001_012    3
A001_013    2
A001_014    8
A001_015    9
A001_016    10
A001_017    9
A001_018    8
A001_019    2
A001_020    3
A001_021    4
A001_022    9
A001_023    5
A001_024    3

Comment: What do you mean "linear data"?
Comment: Clustering is easy but most algorithms offer little control over the cluster sizes.
Looking for structural breaks in the data generating process is harder, especially with such small groups, but possibly the slightly more appropriate approach for what you are doing.
In my personal opinion the problem you are trying to solve is nontrivial.  Hopefully someone has something to offer in the way of a solution.
Comment: @Tim the only way I can describe it is that the sections should be homogeneous. There is no rule per say; however, if I had to force some, I'd say 8-10, 5-7, 2-4, 0-1
Comment: This looks more like changepoint detection!
Answer: A dynamic program to minimize the sum of group variances subject to these constraints is simple and reasonably fast, especially for such a narrow range of group sizes.  It reproduces the posted solution.

The data are plotted as point symbols.  The groups are color-coded and separated by vertical lines.  Group means are plotted as horizontal lines.
Commented R code follows.  It computes the solution recursively, achieving efficiency by caching the results as it goes along.  The program cluster(x,i) finds (and records) the best solution starting at index i in the data array x by searching among all feasible windows of lengths n.min through n.max beginning at index i.  It returns the best value found so far (and, within the global variable cache$Breaks, leaves behind an indicator of the indexes that start each group).  It can process arrays of thousands of elements in seconds, depending on how large the range n.max-n.min is.  For larger problems it would have to be improved to include some branch-and-bound heuristics to limit the amount of searching.
#
# Univariate minimum-variance clustering with constraints.
# Requires a global data structure `cache`.
#
cluster <- function(x, i) {
  #
  # Cluster x[i:length(x)] recursively.
  # Begin with the terminal cases.
  #
  if (i > cache$Length) return(0)                    # Nothing to process   $
  cache$Breaks[i] <<- FALSE                          # Unmark this break    $
  if (i + cache$n.min - 1 > cache$Length) return(Inf)# Interval is too short
  if (!is.na(v <- cache$Cache[i])) return(v)         # Use the cached value $
  n.min <- cache$n.min + i-1                         # Start of search      $
  n.max <- min(cache$n.max + i-1, cache$Length)      # End of search
  if (n.max < n.min) return(0)                       # Prevents `R` errors
  #
  # The recursion: accumulate the best total within-group variances.
  # To implement other objective functions, replace `var` by any measure of
  # within-group homogeneity.
  #
  values <- sapply(n.min:n.max, function(k) var(x[i:k]) + cluster(x, k+1))
  #
  # Find and store the best result.
  #
  j <- which.min(values)
  cache$Breaks[n.min + j] <<- TRUE  # Mark this as a good break $
  cache$Cache[i] <<- values[j]      # Cache the result          $
  return(values[j])                 # Pass it to the caller
}
#
# The data.
#
x <- c(3,2,1,3,4,5,0,0,0,1,2,3,2,8,9,10,9,8,2,3,4,9,5,3)
#
# Initialize `cache` to specify the constraints; and run the clustering.
#
system.time({
  n <- length(x)
  cache <- list(n.min=4, n.max=10,      # The length constraints
                Cache=rep(NA, n),       # Values already found
                Breaks=rep(FALSE, n+1), # Group start indexes
                Length=n)               # Cache size
  cluster(x, 1)           # I.e., process x[1:n]
  cache$Breaks[1] <- TRUE # Indicate the start of the first group $
})
#
# Display the results.
#
breaks <- (1:(n+1))[cache$Breaks]                # Group start indexes $
groups <- cumsum(cache$Breaks[-(n+1)])           # Group identifiers
averages <- tapply(x, groups, mean)              # Group summaries
colors <- terrain.colors(max(groups))            # Group plotting colors

plot(x, pch=21, bg=colors[groups], ylab="Rating")
abline(v = breaks-1/2, col="Gray")
invisible(mapply(function(left, right, height, color) {
  lines(c(left, right)-1/2, c(height, height), col=color, lwd=2)
}, breaks[-length(breaks)], breaks[-1], averages, colors))

Comment: All I can say is Wow! Thank you very much. You've always been such a great help whether on this site or GIS.SE. I really appreciate the thorough and perfect answer
Comment: +1, great answer! I just naively ran your code but changed the data to `x <- (3,2,1,3,4,5,0,0,0,1,2,3,2,8,9,100,9,8,2,3,4,9,5,3)` (everything else is the same). But now, several groups of size 1 are displayed. Do you know why this is the case?
Comment: @COOL That doesn't happen for me.  I stuck the missing `c` in front of the opening parenthesis (so it reads something like `x <- c(3,2,1,3,4,5,0,0,0,1,2,3,2,8,9,100,9,8,2,3,4,9,5,3)`), pasted the whole line in immediately before the `system.time` statement, and re-ran the entire file.  I get four groups starting at positions 1, 5, 15, and 21.  Note that I adopted a very quick and dirty architecture which requires you to re-initialize the `cache` data structure before re-running the code.  (I should have encapsulated that better.)
Comment: You're right! I played around with some numbers and must have forgotten to re-initialize the cache data structure. Sorry.
Comment: @whuber if I want to write a paper and reference your answer to my question in my methodology question and the fact that I'm converting the code into python. What's the correct way to do so?
Comment: I don't know, but for some ideas, visit the [Math Overflow discussion on this topic](http://tea.mathoverflow.net/discussion/7/cite-mathoverflow/).


How do I use the flood interface?

Question: I'm working on a two-factor login form that sends a text message to the user. With a "did not recieve message" button. I would like to limit the number of times a user can request for a new text message.
I know that I can register flood for a certain event like: \Drupal::flood()->register('eventName', 1800, $user->getUuid()). The FloodInterface has a isAllowed($name, $threshold, $window = 3600, $identifier = NULL); method that, I imagine, tests when a user tries to do a certain $name event, it tests if that event has been done more often than the $threshold within the $window. If so then the method returns false and otherwise true.
In my example I can register the event somewhere in the chain of events/functions that send the message and also check there if the event isAllowed?
Something like so:?
public function someFunction() {
  \Drupal::flood()->register('send message', 900, $user->getUuid());
  $token = 12345;
  $this->sendMessage($token);
}

private function sendMessage($token) {
  $is_allowed = \Drupal::flood()->isAllowed('send message', 3, 900, $user->getUuid());
  if ($is_allowed) {
    // Send the message
  }
}

What I don't get is why I also have to indicate the $window in the isAllowed method when I already assign that in the register method?
Answer: They are two different parameters that share the same name (and can share the same value by default). Per the docs for FloodInterface:

::register():

int $window: (optional) Number of seconds before this event expires. Defaults to 3600 (1 hour). Typically uses the same value as the isAllowed() $window parameter. Expired events are purged on cron run to prevent the flood table from growing indefinitely.

::isAllowed():

int $window: (optional) Number of seconds in the time window for this event (default is 3600 seconds, or 1 hour).

The register() parameter specifies how long that flood event will be stored (remembered) after the time of registration before ::garbageCollection() clears it out. The isAllowed() parameter specifies the max number of instances that can occur within that window before isAllowed() will return FALSE
Comment: So I get the difference between the two $window variables. But it's not clear to me if the $itentifier for the `::isAllowed` method will use the same value that you used when registering the even when you do not include it in the `isAllowed::` call.
Comment: It will depend on the implementing class of course, but the default for core flood classes is to use the client IP address when `$identifer` hasn't been specified.

To make things more concrete, take a look @ [`DatabaseBackend`](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Flood%21DatabaseBackend.php/class/DatabaseBackend/9.0.x) class that implements the interface via the `flood` table.

How do I use the flood interface?


Use m4 templating to templetify file name

Question: I have a project that uses m4 templates to templetify things.
What I need to do is to actually create files based on parameters in m4 files. File content will stay the same.
So, let's say I have this:
DEV.m4
define(`A_NODE',`myhostname')
define(`A_NODE_PORT',`1234')

filter.template
aaa=[aaaaa]
bbb=[bbbbb]

I would need to create a file:
filter_%%A_NODE%%_%%A_NODE_PORT%%.cfg with the data from filter.template
Is it possible within m4 or I would need to write some scripting code?
Answer: define(`A_NODE',`myhostname')
define(`A_NODE_PORT',`1234')
syscmd(`cp filter.template filter_'A_NODE`_'A_NODE_PORT`.cfg')

Comment: A little explanation never hurt anyone.


Download XML File with POST Method and Store it in Document's Directory

Question: I was successful on getting a reply from the server with my post method however, I have problem with downloading the xml file data.
Here is my Post method (that I've searched in stackoverflow)
//We begin by creating our POST's body as an NSString, and converting it to NSData.
NSString *post = [NSString stringWithFormat:@"device_unique_key=%@&provision_type=c", deviceUniqueKey];
NSData *postData = [post dataUsingEncoding:NSASCIIStringEncoding allowLossyConversion:YES];

//Next up, we read the postData's length, so we can pass it along in the request.
NSString *postLength = [NSString stringWithFormat:@"%lu", (unsigned long)[postData length]];

//Now that we have what we'd like to post, we can create an NSMutableURLRequest, and include our postData.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] init];
[request setURL:[NSURL URLWithString:@"http://192.168.1.166/autoprovision/sip_setup/downloadSipSetup"]];
[request setHTTPMethod:@"POST"];
[request setValue:postLength forHTTPHeaderField:@"Content-Length"];
[request setHTTPBody:postData];

//And finally, we can send our request, and read the reply by creating a new NSURLSession:
NSURLSession *session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]];
[[session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
    NSString *requestReply = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];

    xmlData = data; //This is the data that I try to donwload as xml file.

    NSLog(@"requestReply: %@", requestReply);
}] resume];

I have xmlData = data //from the dataTaskRequest
Then here is my code for saving the xmlData to the Document's Directory
// Display the network activity indicator
[[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:YES];

// Perform the request on a new thread so we don't block the UI
dispatch_queue_t downloadQueue = dispatch_queue_create("Download queue", NULL);
dispatch_async(downloadQueue, ^{

    NSError* err = nil;
    NSHTTPURLResponse* rsp = nil;

    // Perform the request synchronously on this thread
    NSData *rspData = [NSURLConnection sendSynchronousRequest:request returningResponse:&rsp error:&err];

    // Once a response is received, handle it on the main thread in case we do any UI updates
    dispatch_async(dispatch_get_main_queue(), ^{
        // Hide the network activity indicator
        [[UIApplication sharedApplication] setNetworkActivityIndicatorVisible:NO];

        if (rspData == nil || (err != nil && [err code] != noErr)) {
            // If there was a no data received, or an error...

        } else {

            // Cache the file in the cache directory
            NSArray* paths = NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES);
            path = [[paths objectAtIndex:0] stringByAppendingPathComponent:@"init.xml"];

            [[NSFileManager defaultManager] removeItemAtPath:path error:nil];
            [xmlData writeToFile:path atomically:YES];
            // Do whatever else you want with the data...

             [self loadDataFromXML];
        }
    });

});

The problem is that the xmlData is not working when I parse it using NSXMLParser. I tried to use an xml file and put it inside my project and it parses the xml but the xmlData that I downloaded doesn't get parsed (parse delegates are not called). I think my way of downloading the file is wrong. Can someone help me?
Comment: Did you try to convert the data to a string (using `NSUTF8StringEncoding`) and log that to the console?
Comment: Yes I converted it and got the correct data. It's the requestReply string.
Comment: Then, there must be something wrong with how you setup your `NSXMLParser`. Are you following the official docs or some tutorial?
Comment: @NicolasMiari But do you think I downloaded the xml correctly? I was following this https://github.com/josh--newman/ParserDemo. I edited the code that calls the xml data from documents directory.
Comment: Why don't you copy the console output and paste it here: http://www.xmlvalidation.com to see if you got a text file that passes as correct XML? You mentioned the delgates are not called; did you implement the delegate method `- (void)parser:(NSXMLParser *)parser
parseErrorOccurred:(NSError *)parseError`?
Answer: I found the problem in my project. My codes were correct after all! I didn't see that the code that I followed uses NSCachesDirectory and when I call the xmlData I used NSDocumentDirectory that's why it doesn't get parsed.


icinga2 multi instance display

Question: i have ~30 independent icinga2 systems. All have their own config.
Not i am locking for a dashboard showing problems off all nodes.
I tryed to setup a master server where i add those nodes.
when i do:
<code>icinga2 node list

Node 'portal-sta' (last seen: Thu Dec 17 14:44:22 2015)
    * Host 'BH060'
        * Service 'Ping'
</code>
I get my nodes and all the services on cli.
But icingaweb2 dont display them.
Is there a solution to hava a dashboard (with force check and acknowledge?) without central configuration?
Or is there a better software for this?
Answer: Use "node list" and "node update-config" to generate the objects on the master. Reloading the daemon will populate your central Icinga Web 2 using the DB IDO backend. Once you acknowledge, add a comment or schedule a downtime for these objects, these events are replicated to the satellite nodes and will be executed over there as well.

icinga2 multi instance display


R não cria gráfico corretamente

Question: Estou utilizando R para gerar gráficos de uma table, com dois campos (nome da empresa e quantidade de vendas), ao utilizar barplot, ele não está colocando os valores certos nos campos.
Por exemplo, se as vendas foram 500.000 ele vai fazer no gráfico o maior valor 500, ou seja ele está deixando de lado os milhares.
Como poderia solucionar esse problema? E barplot é a melhor alternativa para gerar gráficos de registros semelhantes a esse:

Nome/Vendas

1     AMAZON/25.268
2      APPLE/743.928
3      CISCO/792.597
4     GOOGLE/780.093
5 HP COMPANY/108.600
6        IBM/144.234
7      INTEL/66.400
8  MICROSOFT/589.300
9    NETFLIX/1.614

Edit: Saída do comando dput(tabela)
structure(list(Nome = structure(1:9, .Label = c("AMAZON", "APPLE","CISCO", "GOOGLE", "HP COMPANY", "IBM", "INTEL", "MICROSOFT","NETFLIX"), class = "factor"), Vendas = c(25.268, 743.928, 792.597,780.093, 108.6, 144.234, 66.4, 589.3, 1.614)), .Names = c("Nome", "Vendas"), class = "data.frame", row.names = c(NA, -9L))

Troquei os pontos por "," e ele passou a dar o seguinte erro:
Error in barplot.default(empresas$Vendas, names.arg = empresas$Nome, main = "Venda por empresa de TI",  :   'height' deve ser um vetor ou uma matriz

Comment: provavelmente o problema está na leitura dos seus dados... edite a sua pergunta colocando o resultado de `dput(tabela)`, para ficar mais fácil de responder!
Comment: O problema está nos seus números, o separador de ponto é decimal no R.
Comment: Quando troquei o ponto por virgula, deu esse erro:
Error in barplot.default(empresas$Vendas, names.arg = empresas$Nome, main = "Venda por empresa de TI",  :   'height' deve ser um vetor ou uma matriz
Comment: @Rafael você não pode colocar vírgula. Ou vai truncar o número ou vai transformar o número em texto.
Comment: @CarlosCinelli como eu posso fazer os números aparecerem corretos no gráfico, sem que eles fiquem como os que você gerou?
Comment: Não entendi sua pergunta, rafael.
Comment: @CarlosCinelli Por exemplo, eu gostaria que aparece-se 600.000 ao invés de 6e+05.
Answer: O seu problema está na leitura dos dados. Não salve o seu csv com ponto como separador de milhares, pois o ponto é separador de decimal no R. Por exemplo, retirando os pontos na mão nos seus dados o barplot funciona normalmente.
empresas <- structure(list(Nome = structure(1:9,
                                            .Label = c("AMAZON", "APPLE","CISCO", "GOOGLE", "HP COMPANY", "IBM", "INTEL", "MICROSOFT","NETFLIX"),
                                            class = "factor"),
                           Vendas = c(25268, 743928, 792597,780093, 1086, 144234, 664, 5893, 1614)),
                      .Names = c("Nome", "Vendas"),
                      class = "data.frame", row.names = c(NA, -9L))

barplot(empresas$Vendas, names.arg = empresas$Nome, main = "Venda por empresa de TI")

Você consegue formatar os labels do eixo y também caso queira, por exemplo:
barplot(empresas$Vendas, names.arg = empresas$Nome, main = "Venda por empresa de TI",
        axes = FALSE)
axis(2, at = c(0, 3e5, 6e5),
 labels = c("0", "300.000","600.000"))

Ou, outro exemplo:
barplot(empresas$Vendas, names.arg = empresas$Nome, main = "Venda por empresa de TI",
        axes = FALSE)
axis(2, at = c(0, 3e5, 6e5),
     labels = c("0", "300 mil","600 mil"))

Comment: rode `options(scipen=5)` antes de fazer o gráfico, assim não vai ficar com a notação científica
Comment: @Rafael, dá para formatar o eixo y sim, inclusive colocando texto, vou colocar um exemplo de como fazer isso
Comment: Mas eu não consigo arrumar o formato dos números?

R não cria gráfico corretamente


Connection with SQL database in visual studio 2010

Question: I have connected with the database successfully by using wizard in server explorer. but the data source is showing nothing. i have googled and search on msdn help. but not get help.
I want to add data source. but add new data source option is disabled for me.
I am using windows form application...
I want ask that am i doing right. could you please tell me how can i get rid of not showing thing.
PLEASE HELP!!!
Answer: The Server Explorer of Visual Studio 2010 and the Datasource of your application are completely different. Check the following links to see how you can connect your Windows Form Application with Sql-Server database:

Walkthrough: Connecting to Data in a SQL Server Express Database (Windows Forms)
Connecting to a Database with Visual Studio Tools
Comment: yes they are absolutely different. but i googled and found the answer.
Comment: Answer: I am using c/C++ windows form application with this there are complexities in it. we can't simply create datasource using it. so I've decided to use the C# ... yes c# is the best ... ans is simple that .. use c# instead of c/c++... and use the wizard to connect to database server and also datasource.


Masked redirect to another folder over root

Question: I have domain www.domain.com, and there are images in www.domain.com/data/photos/*.*. What I want, is access this images (with Virtual Host and masked - ie. I dont want the url to be changed, I want to have virtual subdomain url) as this url http://img.domain.com/*.*
My major website is on /data/web/domain.com/web/*.*.
Subdomains can be created in /data/web/domain.com/sub/*.*.
I tried this, but its not working:
DocumentRoot /data/web/domain.com/sub
ServerName domain.com
ServerAlias img.domain.com
Alias /img /data/web/domain.com/web/data/photos

Also tried about 5 solutions here on SO, but none worked for me.
Where is the problem?
EDITED:
This is my current settings of Virtual Host:
DocumentRoot /data/web/domain.com/web
ServerName www.domain.com
ServerAlias domain.com
ServerAlias img.domain.com

And I tried to add this to .htaccess file:
RewriteEngine on
RewriteBase /

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{HTTP_HOST} ^img.domain.com [NC]
RewriteRule ^(.*)$ http://www.domain.com/data/photos/$1 [R,L]

It is working, but it is changing the url, when I enter f.e http://img.domain.com/image.jpg it is changed in the browser to http://www.domain.com/data/photos/image.jpg.
In short; I want when you type http://img.domain.com/image.jpg to show the content of file located in /data/web/domain.com/web/data/photos/image.jpg.
How to prevent this change of URL?
Answer: You can use
RewriteRule "/(.*)\.(jpg|gif|png)$" "http://img.domain.com/$1.$2" [P]

Instead of:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{HTTP_HOST} ^img.domain.com [NC]
RewriteRule ^(.*)$ http://www.domain.com/data/photos/$1 [R,L]

According to apache RewriteRule documentation
Note: mod_proxy must be enabled in order to use this flag.
EDITED
I have found a solution!
Try this:
RewriteCond %{ENV:REDIRECT_FINISH} !^$
RewriteRule ^ - [L]
RewriteCond %{HTTP_HOST} ^img.domain.com$ [NC]
RewriteRule ^(.*)$ /data/photos/$1 [L,E=FINISH:1]

The first two rows help you to prevent infinite loop.
Comment: I changed it, its working also without `,E=FINISH:1`.
Comment: This will do something really different. Also I dont want to use mod_proxy, I am on the same domain. I want when you type `http://img.domain.com/image.jpg` to show the content of file located in `/data/web/domain.com/web/data/photos/image.jpg`.
Comment: I have updated my post, hope this would help you. Sorry for my first useless answer.
Comment: Of course I tried removing `R`, but it wont help. It will do nothing then.
Comment: I have found it! Please, check my updated post. Hope this time I will help you.
Comment: so, what about my updated solution? I have tested it, it returns something from /data/photos/ without redirecting.


Most optimal way to programmatically check if site is running locally or on a server with Django?

Question: Currently I have this in my settings.py file:
DEBUG = True
LOCAL = True
TEMPLATE_DEBUG = DEBUG

SITE_TITLE = 'Stack Overflow Question'

REMOTE_SITE_URL = "http://************:8080"
LOCAL_SITE_URL  = "http://************:8000"

ADMINS = (
    # ('Your Name', 'email@example.com'),
)

MANAGERS = ADMINS

if LOCAL:
    SITE_URL = LOCAL_SITE_URL
    ROOT_URL = '/mnt/hgfs/Sites/************'
    engine = 'sqlite3'
    dbName = '************.db'
    dbUser = ''
    dbPassword = ''
    dbHost = ''
    dbPort = ''
else:
    SITE_URL = REMOTE_SITE_URL
    ROOT_URL = '/var/www/vhosts/www.*************.com/public/'
    engine = 'mysql'
    dbName = '************'
    dbUser = 'www-data'
    dbPassword = '************'
    dbHost = ''
    dbPort = ''

The idea is that if I modify LOCAL=True to false, I can switch between the sqlite3 database and the mysql that's kept on the server. But this is cumbersome when using SVN. What I want is the ability for settings.py to intelligently know if it's on the server or running locally. Is there a way to sniff the URL, IP or simply check a file that exists on the server and not locally? Not only am I looking for a solution, but one that doesn't tax the server. Checking if a file exists might be a bit heavy of a process if it occurs every time Django renders a page. Any suggestions?
Answer: Checking if a file exists will not occur every time a render occurs.  It will actually only occur whenever you interpreter process is started, which all depends on your deployment configuration.  This will depend on a variety on your webserver setup, but if you are using apache, chiefly MaxRequestPerChild and StartServers, and related parameters.  For example, if I have
StartServers 8
MaxrequestsPerChild 4000

in a preforked apache, I will test that file for the first 8 requests, listen to 32,000 more requests, then test that file 8 more times.  (Yes, I know it's much more complicated than this.)
The point is, most implementations that load your code will not re-source the settings.py file very often.  With that in mind, how about:
import os
if os.uname()[1] == 'my.development.server.com':
    LOCAL = True
else:
    LOCAL = False

Comment: Perfect! Thanks again for the quick response. I think this works out great.


Getting glob to follow symlinks in Python

Question: Suppose I have a subdirectory of symlinks that looks like the following:
subdir/
    folder/
        readme.txt
    symlink/ => ../hidden/
hidden/
    readme.txt

If I run the following code:
>>> from pathlib import Path
>>> list(Path('./subdir/').glob('**/readme.txt'))

I would expect the outcome to be:
subdir/folder/readme.txt
subdir/symlink/readme.txt

But the actual result is:
subdir/folder/readme.txt

I found out that this is because (for some undocumented reason) the ** operator doesn't follow symlinks.
Is there a way to change this configuration pragmatically?
Comment: That's strange, since there's an [open issue](https://bugs.python.org/issue29475) asking for `glob` to optionally *not* follow symlinks.
Comment: This behavior seems to have been caused as a side effect of fixing this issue: https://bugs.python.org/issue26012. The method `_iterate_directories()` in class `_RecursiveWildcardSelector` of pathlib.py explicitly ignores symlinks.
Comment: Probably of interest: https://github.com/python/cpython/issues/77609#issuecomment-1567306837
Answer: pathlib.glob also doesn't work for me with ** and symlinks. I've found related issue https://bugs.python.org/issue33428.
As an alternative for Python3 you could use glob.glob with ** and recursive=True option (see details https://docs.python.org/3/library/glob.html)
In [67]: from glob import glob
In [71]: list(glob("./**/readme.txt", recursive=True))
Out[71]:
['./hidden/readme.txt',
 './subdir/folder/readme.txt',
 './subdir/symlink/readme.txt']

In [73]: list(glob("./**/readme.txt", recursive=False))
Out[73]: ['./hidden/readme.txt']

Compare to:
In [72]: list(Path('.').glob('**/readme.txt'))
Out[72]: [PosixPath('hidden/readme.txt'), PosixPath('subdir/folder/readme.txt')]

Answer: I've never used pathlib before, so you may extend this solution to take advantage of some of its features, but I got this to work using glob only.
from glob import glob
list(glob('./subdir/*/readme.txt'))

Output:
['./subdir/folder/readme.txt', './subdir/symlink/readme.txt']

If you're set on using glob with more than one depth of subdirectory, the hackish solution is to include variations with extra */ (e.g. ./subdir/*/*/*/readme.txt) up to some arbitrary depth, and concatenate the results from each variation.
The more appropriate way to do what you want would be to write a custom function that has the behavior you want (searches through symlinks to arbitrary depth), and handles the case of circular paths in the way you want. See this question for tips on doing this with os.walk (remember to set followlinks=True).
Comment: `Path.glob` and `glob` use the same functionality - the issue I have found is that glob doesn't like following symlinks when using `**`.
Comment: Is there a reason you can't use `*` instead of `**`, since your directory structure has a specific depth? Interesting to note that [`glob` in javascript](https://www.npmjs.com/package/glob) has the same behavior for `**`.
Comment: I am guessing the reason for this behavior is to avoid issues with infinite depth in the case of symlinks that create directory loops.
Comment: I'm trying to write a solution abstract enough that it can be used in a few case scenarios. There will be instances where the readme.txt file may be burried deep
Comment: Updated solution to reflect your need for deeper paths.
Answer: python 3.6 works for me with a call to rglob,
import pathspec

p = pathspec.Path("./subdir").rglob("readme.txt")


How to install phpUnit in Symfony4 / Symfony5?

Question: When I install phpUnit directly with composer req phpunit/phpunit I get a warning message:
Adding phpunit/phpunit as a dependency is discouraged
in favor of Symfony's PHPUnit Bridge.

  * Instead:
    1. Remove it now: composer remove --dev phpunit/phpunit
    2. Use Symfony's bridge: composer require --dev phpunit

So I do as the message says, but after composer require --dev phpunit it installs the symfony/test-pack which only install the Panther and the phpUnit bridge, but the phpUnit itself is nowhere to be seen.
How to install phpUnit in Symfony4 properly?
Answer: I ran the vendor/bin/simple-phpunit script by accident, and turns out it downloads the phpUnit library locally when executed.


What is fundamentally different between a logistic regression and a Logit Discrete Choice model?

Question: I'm trying to understand the difference between the two. I know there is a background random utility theory in discrete choice modeling, but I can't pinpoint, in what I've read, the difference between the logit choice model (sometimes called a conditional logit choice model, to differentiate from a nested choice model).
Maybe it is something related to the likelihood equation, but I'd love some more technical clarification.
Comment: Possibly important is the link between the [Gumbel distribution](https://en.m.wikipedia.org/wiki/Generalized_extreme_value_distribution#Link_to_logit_models_(logistic_regression)) and the logistic distribution and categorical distribution. The logit choice model seems to be derived using a different latent variable model (but ending up with the same as logistic regression).
Comment: What do you mean by "nested choice model"?
Comment: Like Alecos, I do neither see it clearly described which regression and which model you are exactly comparing.
Answer: Logistic regression (using logit link) is a binary response model, with $P(y_i=1|x_i)=\hat\pi_i$ and $\hat\theta_i=x^T_i\hat\beta=logit(\hat\pi_i)$. Whether we want to predict $P(y_i=1|x_i)$ or $P(y_i=0|x_i)$, we use the same covariates for prediction. If the response has more than 2 levels, then the usual GLM model would be multinomial regression.
In a way, DCM is the far more complex version of logistic regression. For each choice $1,...,k$ we have a set of corresponding covariates, and naturally enough all other covariates are not used by this choice. That is, there is some kind of hierarchical structure of the covariates. The set of variables belonging to person $i$ corresponding to alternative $j$ is denoted $z_{ij}$ and then we can write the utility $U_{ij}=z_{ij}^T\beta+\epsilon_{ij}$, and in general we expect person $i$ to choose alternative $j$ if $U_{ij}>U_{ik},\forall k\ne j$.
If we look at DCM for two alternatives, then it can be easily be written as logistic regression (see here). If we have $K\ge3$ alternatives, then the probability of person $i$ choosing alternative $j$ is $P_{ij}=\frac{\exp(\beta z_{ij})}{\sum_{k=1}^K\exp(\beta z_{ik})}$. This is also sometimes called multinomial logit.
Therefore, logistic regression is the binary special case of the multinomial logit model.


Angular2 [disabled]="function()" doesn't work in IE

Question: In Angular2 app the submit button is enabled only if several conditions are met. I use [disabled]="disabled(j)" for that.
html:
<button type="submit" class="submit" [disabled]="disabled(j)">Submit</button>

component:
disabled(j) {
   if (this.form.valid && j === this.userId) {
    return false;
   } else {
    return true;
   }
}

It works perfectly in Chrome, but doesn't work in IE. Inspecting DOM element has shown, that instead of [disabled]="disabled(j)" it has [disabled]=""
How to fix that?
Answer: Most likely problem is with polyfills for form validation (which uses HTML5 and ES6 standards), and not actually the function inside an input directive. More specific, this item is the likely culprit: this.form.valid
Check out how to add polyfills, and which extra packages will be needed: https://angular.io/guide/browser-support
Note: IE is considered a deprecated browser in most applications, and full support should be avoided as a requirement (especially for versions < 10). For 10/11 the ES6 polyfills are still required.
Comment: I'm concerned that IE didn't bind a function to disabled property - `[disabled]=""` so it's not hitting this.form.valid check.
Comment: @AnnaOlshevskaia - You can check that with a `console.log` statement at the start of the `disabled(j)` method.
Answer: As a work around, you can simply add the condition directly to [disabled] attribute
<button type="submit" class="submit" [disabled]="!(form.valid && j === userId)">Submit</button>

Answer: This may have to do with polyfills, since you are using IE you may have to manually update them. See here: Angular 2 / 4 / 5 not working in IE11


MediaPlayer replays when other MediaPlayer starts

Question: So in my program I have a recyclerview with a series of buttons and when a button is clicked it should play a sound file. But if there are any other sounds playing it should stop them first and play the current sound.
However, when I select one and quickly select another one it will replay the first one and then play the second one.
Edit: so im noticing that when the first one replays, it stops relatively quickly, so it doesnt play the entire file, but it still starts
recyclerView.addOnItemTouchListener(
            new RecyclerItemClickListener(this, new RecyclerItemClickListener.OnItemClickListener() {
                @Override
                public void onItemClick(View view, int position) {
                    for(MediaPlayer player : mediaPlayerList){
                        if(player.isPlaying()){
                            player.pause();
                            player.seekTo(0);
                        }
                    }

                    mediaPlayerList.get(position).start();

                }
            })
    );
}

Everything works fine, its just the first sound selected plays twice for some odd reason, I placed a pause() and seekTo(0) so that it would stop it and allow it to be replayed from the beginning
Im not sure if this is related but I am also getting a logcat error :  getSlotFromBufferLocked: unknown buffer: 0xab80f170.
MediaplayerList contains
final List<MediaPlayer> mediaPlayerList = new ArrayList<MediaPlayer>();
        mediaPlayerList.add(0,mediaPlayer0);mediaPlayerList.add(1,mediaPlayer1);
        mediaPlayerList.add(2,mediaPlayer2);mediaPlayerList.add(3,mediaPlayer3);
        mediaPlayerList.add(4,mediaPlayer4);mediaPlayerList.add(5,mediaPlayer5);

each MediaPlayer
final MediaPlayer mediaPlayer0 = MediaPlayer.create(this, R.raw.sound);

Answer: Rather than doing player.pause() if a song is already playing and you select another song, you need to stop and reset the media player like
Instead of
player.pause();
player.seekTo(0);

Do
player.stop();
player.reset();

Edit: Try doing something like this. Make a global media player object like
private MediaPlayer player;

Then for playing the song do
recyclerView.addOnItemTouchListener(
        new RecyclerItemClickListener(this, new RecyclerItemClickListener.OnItemClickListener() {
            @Override
            public void onItemClick(View view, int position) {
                if(player.isPlaying()){
                     player.stop();
                     player.reset();
                }
                player = mediaPlayerList.get(position);
                player.start();
            }

Edit 2: OK try this
recyclerView= (RecyclerView) findViewById(R.id.meme_list);
adapter = new Adapter(this,getData());
recyclerView.setAdapter(adapter);
recyclerView.setLayoutManager(new LinearLayoutManager(this));
player.setOnPreparedListener(new MediaPlayer.OnPreparedListener() {
        public void onPrepared(MediaPlayer player) {
            player.start();
        }
});
recyclerView.addOnItemTouchListener(
            new RecyclerItemClickListener(this, new RecyclerItemClickListener.OnItemClickListener() {
                @Override
                public void onItemClick(View view, int position) {

                       if (player.isPlaying())
                       {
                           player.stop();
                           player.reset();

                       }

                    player= mediaPlayerList.get(position);
                    player.prepare();

                }
            })
    );

Comment: After about 8 different starts, im getting logcat errors saying "start called in state 1", "error (-38, 0)"
Comment: i can select other files, but then they too eventually stop working and producing the same error
Comment: You also need to setDataSource and call player.prepare() before you start the media player
Comment: Check out the activity cycle of media player in android here http://developer.android.com/reference/android/media/MediaPlayer.html . The data source in the media player will not be cleared when you stop the media player. For clearing the previous song's data source you need to call reset()
Comment: is there an alternative to setDataSource? the data source is determined based on which list item is clicked.
Comment: Does your mediaPlayerList contain the Uri of the song?
Comment: Can you show me how you are playing the first song? and what mediaPlayerList actually contains?
Comment: I updated the post, im not sure what you mean by how i am playing the first song.  It starts when clicked from the recyclerview
Comment: player.prepare() is highlighted red as for unhandled IO excpetion
Comment: Actually get rid of prepare() since you are using create to start music you don't need to use prepare()
Comment: player returned a fatal null pointer exception, i tried adding a for each loop to fix it but resulted with the same problem
Comment: Did you create new MediaPlayer? Like private MediaPlayer player = new MediaPlayer();
Comment: fixed that, and reverted back to what you suggested.  Still getting error (-38,0)
Comment: based on the flow chart from the android reference it seems to me that setDataSource() -> prepare() is the only way... would there be a way to get the data source within the method?
Comment: Or could prepare async be used and then from there onpreparedlistenere.onprepared
Comment: You can use create for raw files. I think I need your full code bro. setDatasource() is needed if you have the song uri. And when you use setDatasource() you need prepare()
Comment: how can i send this to you? i could maybe email?
Comment: Post the entire class where you are doing anything with mediaPlayer here itself
Comment: posted the entire class
Comment: After a sound is played more than once it produces these errors:
 E/MediaPlayer: start called in state 0
 E/MediaPlayer: error (-38, 0)
Comment: What does it say on the logcat?
Comment: E/MediaPlayer: start called in state 0
error (-38,0)
Comment: player.prepare() is still being highlighted as an unhandled IO exception
Comment: I used a try catch statement, but it produced a fatal exception, prepare() is causing issues
Comment: Try to be descriptive please, what issues?
Comment: It seems as if player.prepare() can not be used with player.setDataSource
Comment: Fatal exception on main, IllegalStateException with prepare()


How can I limit/stop the number of characters fgets reads?

Question: I need to my code to only read words of 10 characters and no longer. The purpose of the code is to detect anagrams. How can I limit fgets to only read 10 characters and after that display a message printf("Word is longer than 10 characters please try again"). My code is below I'd appreciate any help.
<code> #include <stdio.h>
    #include <string.h>
    #include <ctype.h>
    #include <stdint.h>

    // Sorting function to sort words

    void lowercase(char *str);
    void removespaces(char *str);
    int lol(char a[], char b[], int *_lenword1, int *_lenword2);

    int main(void){

    //Initiating two strings with a maximum limit fo 10 characters
     char FirstWord[12];
     char SecondWord[12];

    /* Console requests input of the first word, fgets reads word of unknown legnth without reading more than buffer allows
       Index begins at 0 to use strlen excess allocated characters can be replaced with /0                                      */
        printf("Please enter the first word: \n");
        fgets(FirstWord, 12,stdin);
        FirstWord[strlen(FirstWord)-1]='\0';

        printf("Please enter the second word: \n");
        fgets(SecondWord, 12, stdin);
        SecondWord[strlen(SecondWord)-1]='\0';

        int lenword1 = strlen(FirstWord);
        int lenword2 = strlen(SecondWord);

     //Relaying strings to functions to convert all characters to lowercase
       lowercase(FirstWord);
       lowercase(SecondWord);
    // Relaying strings to removespaces function so spaces in string are removed
       removespaces(FirstWord);
       removespaces(SecondWord);
    //Decisions
       if (lol(FirstWord, SecondWord, &lenword1, &lenword2))
        printf("%s and %s are anagrams.\n",FirstWord, SecondWord);
      else
        printf("%s and %s are not anagrams.\n", FirstWord, SecondWord);

      return 0;
    }

    //Function to sort characters
    int lol(char a[], char b[], int *_lenword1, int *_lenword2){
    if (*_lenword1 == *_lenword2){
    int first[26] = {0}, second[26] = {0}, c=0;

      // Calculating frequency of characters of the first string

      while (a[c] != '\0') {
        first[a[c]-'a']++;
        c++;
      }

      c = 0;

      while (b[c] != '\0') {
        second[b[c]-'a']++;
        c++;
      }

      // Comparing the frequency of characters

      for (c = 0; c < 26; c++){
        if (first[c] != second[c]){
          return 0;
        }
      }
      return 1;
    }
    else{ // if different lengths - then they're not anagrams
            return 0;
        }
    }
    // Function to convert all letters to lower case (to allow for case sensitivy anagram detection)
    void lowercase(char* str){
    for (uint8_t i=0; str[i]; i++ ){
       str[i]=tolower(str[i]);
    }
    }
    // Function to remove all spaces from a given string
    void removespaces( char* str)
    {
       const char* d = str;
        do {
            while (*d == ' ') {
                ++d;
            }
        } while (*str++ == *d++);
    }
</code>
Comment: It's usually easiest to read words of arbitrary length, then call `strlen`, then complain if it's too long.
Comment: `FirstWord[strlen(FirstWord)-1]='\0';` is bad. What happens if `fgets` *fail*? Use `FirstWord[strcspn(FirstWord, "\n")] = '\0';` instead, *after* making sure that `fgets` didn't fail.
Comment: @Someprogrammerdude `FirstWord[strlen(FirstWord)-1]` also a hacker exploit when `fgets()` _reads_ an initial null character.
Comment: Rather than limit input to 10, allow generous, like 100, input and _detect_ overlong input .  User input is  evil.
Comment: As for your problem, how about something like `int res = scanf("%10s%c", FirstName, &temp); if (res == EOF || res == 2 && !isspace(temp)) { /* Error, or input too long */}`
Answer: Something like this:
<code>#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main()
{
    char v[12];

    //Loop checks if fgets succeeded and if there is '\n' in the string:
    while (!fgets(v, 12, stdin) || strcspn(v, "\n") == 11){
        if(feof(stdin)) {
             printf("Error: End of file reached.\n");
             exit(EXIT_FAILURE);
        }
        printf("Error: The word is longer than 10 characters, try again.\n");
        for(int ch=getchar(); ch != '\n' && ch != EOF; ch=getchar());
    }
    v[strcspn(v, "\n")] = '\0';

    printf("%s\n", v);

    return 0;
}
</code>
If there isn't a '\n' in the string then the word has more than 10 characters.
Comment: Lots of good improvements. UV.  For deeper review and to get rid of other corner weaknesses, consider https://codereview.stackexchange.com/
Comment: @chux-ReinstateMonica If `fgets` returns `NULL` the loop continues.
Comment: @chux-ReinstateMonica I've changed it.
Comment: @chux-ReinstateMonica Here it is:  https://codereview.stackexchange.com/questions/256166/limit-fgets-input-to-up-to-a-certain-number-of-characters


If correlation doesn't imply causality then what's the value of knowing the correlation between two variables?

Question: Let's say as a business owner (or marketing or anyone who understands a scatter plot) is shown a scatter plot of two variables: number of advertisements vs number of product sales per month for the past 5 years (or another time-scale so that you have more samples. I just made this one up).
Now he/she see's the scatter plot and is told that the correlation coefficient (corr) is:

1 or
0.5 or
0.11 or
0 or
-0.75 or
-1

Basically any valid value for corr
Question: What does this even mean to a decision maker or any consumer of the scatter plot? What decisions can one take just based on this?
I.e.: What is the use of seeing correlation between any two variables and what can one do with that information in isolation? Is it only to see what to and not to consider for inclusion in regression analysis or is there a more practical use?
Just curious, I've always worked with this technique, but I've been told that correlation by itself is not of much use - so what "IS" the use?
Answer: The phrase "correlation doesn't imply causation" gets overplayed.  (As Cohen wrote, "it's an awfully big hint".)  We beat this phrase into students because of a bias intrinsic to the human mind.  When you hear 'the crime rate is correlated with the poverty rate', or something like that, you cannot help but think this means that poverty causes the crime.  It is natural for people to assume this, because that's the way the mind works.  We use the phrase over and over in the hopes of counteracting that.  However, once you've absorbed the idea, the phrase loses most of it's value, and it's time to move on to a more sophisticated understanding.
When there is a correlation between two variables, there are two possibilities: it's all a coincidence, or there's some causal pattern at work.  Calling a pattern in the world a coincidence is a terrible explanatory framework and should probably be your last resort.  That leaves causality.  The problem is that we don't know the nature of that causal pattern.  It could well be that poverty causes crime, but it could also be that crime causes poverty (e.g., people don't want to live in a high-crime area, so they move out and property values fall, etc.).  It could also be that there is some third variable or group of variables that cause both crime and poverty, but that there is, in fact, no direct causal link between crime and poverty (known as the 'common cause' model).  This is especially pernicious, because, in a statistical model, all other sources of variation are collapsed into the dependent variable's error term.  As a result, the independent variable is correlated with (caused by) the error term, leading to the problem of endogeneity.  These problems are very difficult, and should not be taken lightly.  Nonetheless, even in this scenario, it is important to recognize that there is real causality at work.
In short, when you see a correlation, you should think that there probably is some sort of causality at play somewhere, but that you don't know the nature of that causal pattern.
Answer: A few thoughts:

The old canard about correlation not being causation is only half the story. Correlation may not be causation, but some form of association between the two variables is a necessary step along the path to showing causation, and correlation can help show that.
It helps point out trends. Show it to a business owner, and they may say "Yeah, that makes sense, you see Widget X and Widget Y both end up being used by a particular group of people, even though they're not really related. Or they might say "that's...odd", at which point you prompted further investigation.
Look at it this way. Correlation is a tool. A hammer, by itself, isn't all that useful. It certainly won't build a house all by itself. But have you ever tried to build a house without a hammer?
Comment: In your first bullet you say that correlation is a necessary condition for causation - that's not true. If there's a non-monotonic relationship between two variables then they can be uncorrelated - this does not preclude causation.
Comment: @Macro - true, and edited
Comment: @Macro True, but in practice you can apply a function to your variable to make the relation to be tested monotonic. if you don't know this function, then... you don't know much about what you are looking for
Comment: @EpiGrad: Assume the X-Y correlation graph of two variables look like a happy smiley (or any other shape per se). The correlation coefficient would actually be quite small, but there certainly would be some inter-relationship, right? How/what should one do in such a case?
Comment: @Nupul A somewhat more complicated exploration of X-Y beyond linearity.
Answer: I thought I was knowledgeable about these things, but it was only last month that I looked up "imply" in the dictionary and found it had two strikingly different meanings.  1. Suggest and 2. Necessitate. (!)  Correlation seldom necessitates causation, but it certainly can suggest it.  As @EpiGrad points out, it is a necessary though not sufficient condition for establishing causation.
As time goes on one hopefully finds a middle ground between seeing correlation as the end-all and as completely useless.  And one takes into account subject-/domain-/content-specific knowledge in interpreting correlational results.  Few people would question there being at least some causal link when seeing the advertising-sales results you describe.  But it's always good to stay open to other possibilities, other variables that could at least partly explain the relationship observed.  Readings about confounding variables, validity, and the like pay off with big dividends.  For example, Cook and Campbell's old classic Quasi-Experimentation has a good section on validity and threats to validity.
Comment: As I pointed out to @EpiGrad, correlation is *not* a necessary condition for causation. There is a widespread conception in data analysis that a relationship between variables always refers to a monotonic relationship, which is tacitly assumed by suggesting that correlation is a necessary condition for causation.
Comment: Fair enough.  Let's say "statistical association" is necessary, then.
Answer: Look at it from a gambling perspective.  Let's say we know that on average people who wear workboots to work will have 1.5 injuries on the job, and people wear loafers will have .05 injuries on average.  Or, maybe the chances of an injury for a person wearing workboots is .85, and the chances of injury to a person wearing loafers is .1.
If I randomly select a person from the population, and tell you the person is wearing workboots, and offer you an even money bet on whether or not they had a workplace injury last year, would you take the bet? Well, you'd take the bet if you were to be able to bet on the side that they had an injury.. 85% of the time you'll win, and you're getting even money.
The point is, knowing that piece of information gives us information about whether or not they are likely to experience an injury at work.. The shoes have nothing to do with it, in fact, the workboots prevent injury.. But the confounding variable here is the type of job that goes along with the workboots.. And maybe other things like the person possibly being more reckless.
Answer: A correlation coefficient, as other measures of association, is useful if you want to know how much knowing the value of X is informative about the value of Y. This is different from knowing whether if you were to set X to a particular value, what value of Y you would get (which is the essence of a counterfactual interpretation of causation).
Nevertheless, in many contexts (e.g. prediction) inferences based on correlation would be valuable in their own right. Yellow teeth are correlated with lung cancer (as they are both probabilistically caused by cancer). There is no causation between the two: whitening teeth would not cure lung cancer. But if you need a quick screening test for who is likely to have lung cancer, checking for yellow teeth might be a good first step.
It is a different question whether the correlation coefficient is the best available measure of association, but I think the question is more about what is the value of knowing non-causal association.
Btw, not only is correlation not sufficient demonstration of causation, but it is not necessary neither. Two variables can be causally related yet exhibit no correlation in any particular dataset (e.g. due to selection bias or confounders).
Answer:
correlation by itself is not of much use - so what "IS" the use?

Let me disagree with this phrase, correlation let to know the level of association between 2 variables. Then, it is useful when trying to explain relation between such variables.
On the other hand, (as Macro wrote) correlation is not a necessary condition for causation, however, is enough to explain the level of association. Furthermore, you can test the independence of the variables, but correlation can give you another useful information, the coefficient of determination.
Nevertheless, analyst must know the domain to be able to explain the kind of relation.
Comment: I'm not sure what you mean by this: `Furthermore, you can test the independence of the variables, but correlation can give you another useful information, the coefficient of determination`
Comment: What I mean was: "you can test the independence of the variables" but anyway, even when not testing the independence, the information of correlation and the coef. of determination are "useful" to understand and to explain the kind of relation between the variables.
Answer: I think data collection and study design may also play a role in answer this question. You won't design a study and collect a set of data completely irrelevant to each other, even in the observational studies.Therefore "the correlation does not imply causation" may be justified. Even though it is not causal relationship, there might be an association related.
However if you are talking about two dataset completely irrelevant, but you still wanna use correlation to explain the association and causation, then it might be inappropriate. For example, if two dataset all have downward trends, say ice cream sales and number of marriages, the correlation coefficient might be very high. But is it necessary to mean an association?
Answer: In addition to Fomite's answer, I include an option that has not been considered in the rest of the answers: correlation is a measure of strength of a relationship when you already know there is causation.
For example, we know that there is a causal relationship between specific humidity in the atmosphere and probability of precipitation at close locations. However, we do not know (or cannot measure) all factors involved, so this relationship is not deterministic. Correlation helps measuring the strength of this association*, e.g. obtaining a spearman correlation of 0.8 says that most of the time an increase in one variable when the other one increases.
*Correlation is probably not the best measure of association, but this is out of the scope of this question.


Tabela em Postgre campo ID não é com auto incremento

Question: vou explicar meu problema:
No meu SQL no sistema já veio o erro no INSERT, informando que o campo id, que é autoincremento está informando que é null:
INSERT INTO cliente_sub_grupo (id,id_cliente,subgrupo,tipo_atendimento,ind_valor_titular_igual_contrato,ind_valor_dependente_igual_contrato,valor_titular,valor_dependente)  VALUES (:id,:id_cliente,:subgrupo,:tipo_atendimento,:ind_valor_titular_igual_contrato,:ind_valor_dependente_igual_contrato,:valor_titular,:valor_dependente)

INSERT INTO cliente_sub_grupo (id,id_cliente,subgrupo,tipo_atendimento,ind_valor_titular_igual_contrato,ind_valor_dependente_igual_contrato,valor_titular,valor_dependente)  VALUES (:id,:id_cliente,:subgrupo,:tipo_atendimento,:ind_valor_titular_igual_contrato,:ind_valor_dependente_igual_contrato,:valor_titular,:valor_dependente)

org.postgresql.util.PSQLException: ERROR: null value in column "id" violates not-null constraint
        Detalhe: Failing row contains (null, 12, vcxvxc, F, t, t, 1, 1, ativo).

Depois de olhar o código resolvi olhar diretamente no BD e na imagem a seguir eu faço um cadastro teste, deixando o campo ID como em branco e dá o erro da segunda imagem:

Este é o erro quando insiro diretamente do BD (o mesmo erro no console de minha aplicação no INSERT):

O mesmo erro de console:

cliente-sub-grupo.insert org.postgresql.util.PSQLException: ERROR: null value in column "id" violates not-null constraint
  Detalhe: Failing row contains (null, 12, vcxvxc, F, t, t, 1, 1, ativo).
org.postgresql.util.PSQLException: ERROR: null value in column "id" violates not-null constraint
        Detalhe: Failing row contains (null, 12, vcxvxc, F, t, t, 1, 1, ativo).

O que pode ser e como consertar este erro.
Answer: Amigo, se o campo id é autoincrement então não precisa declará-los no seu insert.
Por exemplo, imagina que você tem uma tabela frutas:
CREATE TABLE frutas(
   id SERIAL PRIMARY KEY,
   nome VARCHAR NOT NULL
);

E o insert ficaria assim:
INSERT INTO frutas(nome) VALUES('laranja');

Repare que não informei o campo id no insert.
No seu caso teria que ser assim:
INSERT INTO cliente_sub_grupo (id_cliente,subgrupo,tipo_atendimento,ind_valor_titular_igual_contrato,ind_valor_dependente_igual_contrato,valor_titular,valor_dependente)  VALUES (:id_cliente,:subgrupo,:tipo_atendimento,:ind_valor_titular_igual_contrato,:ind_valor_dependente_igual_contrato,:valor_titular,:valor_dependente)

Comment: Não estava no INSERT o erro. Este Insert que postei era do console de uma gambiara de uma outra empresa que passou este job de uma biblioteca chamada Trust. Fazendo o teste direto no banco de dados (INSERT) mesmo deixando o campo id em branco, dava o mesmo erro. Era erro na tabela, que foi corrigido como mencionei a cima.
Answer: @Murilo Portugal
Já resolvi, fiz tudo isso que sugeriu e outros testes aqui, mas o erro é que no campo id na coluna Default, como na imagem a seguir, faltou o:
nextval('pessoa_id_pessoa_seq'::regclass)

E funcionou.
Comment: @Ramos, exatamente como o amigo acima disse, para utilizar autoincrement utilize o SERIAL no postgre, desse modo fica implícito, na minha resposta considerei que você já estivesse utilizando ele, que é o mais comum. Mas de qualquer forma que bom que deu certo
Comment: Se você declarar o campo como SERIAL então isto estará implícito. Veja a explicação do tipo de dado SERIAL em: https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-SERIAL


Proving convexity of $f(K)=\ln\left(\frac{\sin(t\sqrt{K})}{\sin(\sqrt{K})}\right)$ for $K\in(0,\pi^2)$

Question: Define for $K\in(0,\pi^2)$ the function
$$f:K\quad\mapsto\quad\ln\left(\frac{\sin(t\sqrt{K})}{\sin(\sqrt{K})}\right)\qquad t\in(0,1)\mbox{ fixed}.$$I need to prove convexity of this function as function of $K$ but how? As function of $K$, the second derivative is pretty terrifying.
Comment: Do you mean convexity in the derivative sense? Or in the sense of: https://en.wikipedia.org/wiki/Convex_function? Either way just use the respective definitions. There will be cancellations in the derivative. Also: it is not obvious whether this is a function of one or two variables.
Comment: I tried the usual definition of convexity but I didn't get anywhere, so I tried the derivative characterization but I got lost (I'm reading a paper where the authors just say "it's convex")

P.s. I didn't checked for differentiability or something, just trying different things
Comment: Just use the derivative definition even if it is "pretty terrifying". You can also use a software as an aid given that you know what you are doing.
Comment: Or you can separate the ln(ab)=ln(a)+ ln(b) or expand the ln(y) and maybe the sin(y) into a series or into their complex definitions etc..
Comment: Yeah I mean, with a calculator you can see clearly that for $K\in(0,\pi^2)$ the second derivative of the thing is greater than $0$, just wanted to check the convexity without the second derivative
Answer: $$f(K)=\log \left(\csc \left(\sqrt{K}\right) \sin \left(\sqrt{K} t\right)\right)=\log \left(\csc \left(\sqrt{K}\right) \right)+\log \left(\sin \left(t\sqrt{K} \right)\right)$$
$$f''(K)=\frac{\sqrt{K} \left(\csc ^2\left(\sqrt{K}\right)-t^2 \csc ^2\left(t\sqrt{K}
   \right)\right)-t \cot \left(t\sqrt{K}\right)+\cot \left(\sqrt{K}\right)}{4
   K^{3/2}}$$
Expanding as a series around $t=0$ gives
$$f''(K)=\frac{\sqrt{K} \cot \left(\sqrt{K}\right)+K \csc ^2\left(\sqrt{K}\right)-2}{4
   K^2}-\frac{t^4}{90}+O\left(t^6\right)$$ It is quite simple to show that the first term is always positive.
Moreover, for small values of $K$
$$f''(K)=\frac{1}{90}+\frac{2 K}{945}+\frac{K^2}{3150}+\frac{4
   K^3}{93555}+O\left(K^{4}\right)$$ and for $K$ close to $\pi^2$
$$f''(x)=\frac{1}{\left(K-\pi ^2\right)^2}+\frac{4 \pi ^2-33}{48 \pi ^4}+\frac{\left(21-2 \pi
   ^2\right) \left(K-\pi ^2\right)}{16 \pi ^6}+O\left(\left(K-\pi
   ^2\right)^2\right)$$


What is the equivalent of a Button Click in DevExpress?

Question: In WPF, a button and a click event is:
<Button Click="Btn_Click" />

private void Btn_Click(object sender, RoutedEventArgs e)
{
   ....
}

In DevExpress, according to the documentation, I would guess the equivalent would be:
<dxb:BarButtonItem ItemClick="Btn_Click" />

private event ItemClickEventHandler Btn_Click(Item sender, RoutedEvent e)
{
   ....
}

But it is not compiling. I would guess it is very easy but I cannot find any example in the documentation.
Answer: The second parameter of the handler should be a ItemClickEventArgs:
private void Btn_Click(object sender, ItemClickEventArgs e)
{
   ....
}

Comment: Sorry, I missed the other mistakes in your code so I didn't fix them... the first parameter must be `object`, and the return type should be `void`. What you saw in the documentation is the declaration of the event, not the signature of the handler...
Comment: I'm porting my XAML to DevExpress XAML, does that mean all the event handler and parameters have to be changed, or does the original event handler still work?
Comment: No, it doesn't work. "Item" does not exist in the current context, "sender", ) expected, and "ItemClickEventArgs" could not be found.


Implantar App Node.js em localhost(Produção)

Question: Desenvolvi uma aplicação com Angular + Node.js.
 O único detalhe é que não há um servidor centralizado(Node.js), ou seja, cada cliente vai ter a sua versão do servidor rodando em sua rede.
 Devido a isso, tive de criar formas de instalar o app em Node.js de modo fácil em cada cliente, testei uma alternativa que não teve o retorno esperado:

Instalar o App como serviço do windows

Onde fiz um .bat com os comandos para instalação de dependências, adição das portas no firewall, instalação dos serviços, etc..
Para fazer isso usei a lib "node-windows" : https://github.com/coreybutler/node-windows
Porém, na maioria dos computadores não executa o script. É preciso executar de novo, as vezes acusa erro de permissão, em outras aparece no console que deu certo, mas na hora de conferir os serviços. eles não estão lá, enfim.. Fiz vários testes, alterações e não resolveu.
Diante disso, gostaria de saber:

Quais alternativas eu tenho para instalar o App em (Node.js) em máquinas locais?(Estou falando em produção, não em testes ou DEV).
Existe alguma forma transparente ou simplificada("Gerar um executável"?! Talvez)
Li em alguns locais que é utilizado o IIS para hospedar a aplicação, alguém já teve experiência nesse sentido, tem algum exemplo do funcionamento?
Comment: Já testei com IIS. Não fica muito bom não, tem que instalar uma coisarada pra funcionar. Acho que uma boa forma de fazer o que você quer é criar um [`docker`](https://docs.docker.com/docker-for-windows/) com tudo que você precisa e apenas executa-lo no cliente.
Answer: Se sua aplicação utiliza Node + Angular, uma possibilidade é a geração de um executável via Electron.
Ele já vem com o Node.js embutido. Entretanto, você pode utilizar pacotes npm e gerar instaladores contendo todos os módulos em uso, com a vantagem que o usuário não precisa instalar nenhum browser específico (já que a engine do Electron é por si só uma versão do Chrome).
Comment: Bacana @OnoSendai, Eu já tinha ouvido falar do Electron, mas não havia pensando nessa possibilidade(nem sabia que dava pra fazer). Porém como ficaria a estrutura do projeto, jogaria os dois projetos que tenho(Cliente e Servidor) dentro de um novo projeto electron? Se não for pedir demais, poderias mostrar um exemplo básico de uso?
Comment: @Geferson em vários projetos que implementei, essa foi a escolha - tanto o conteúdo estático (cliente) quanto o back-end eram embutidos na mesma solução. Um excelente exemplo pode ser estudado aqui: https://scotch.io/tutorials/creating-desktop-applications-with-angularjs-and-github-electron


Why Is ACCESS_ONCE so complex?

Question: The Linux ACCESS_ONCE macro is defined as follows:
#define ACCESS_ONCE(x) (*(volatile typeof(x) *)&(x))

I understand what this does, but wondered why it is so complex ? My understanding is it does the following:

Take address (create a temp pointer to) the variable of concern
Cast this to a volatile pointer of the same type
De-reference that pointer

Any ideas why this isn't implemented in a simpler fashion, say:
 #define ACCESS_ONCE(x) ((volatile typeof(x))(x))

Answer: The ACCESS_ONCE macro is used in situations where a value is being retrieved from a storage location that is known (or suspected) to be volatile, but is not typed as such. The purpose is to retrieve the current value of the storage location in such a way as to defeat an optimising compiler that might otherwise cache the value in a register, or not even provide a storage location at all.
The construct as written applies the 'volatile' to the storage location indirectly by declaring a suitably typed pointer to that location. As per the C standard, this mandates that the object be evaluated strictly according to the rules of the abstract machine.
Your proposed modification would not apply volatile to the storage location, so it would not achieve this purpose. The value retrieval could be subject to optimisation.
Incidentally, I regard this as a model of conciseness for the stated purpose. I reserve complex for far worse things than this.
Comment: Thank you, David, for your clear & precise explanation.
Comment: Some more context: the C standards allow to assign the `volatile` qualifier to *data*, not to the *accesses* to the data. There is a good quantity of people who consider that this is stupid; among them, Linus. Additionally, `volatile` has some more problems, to the point that it's kinda-considered-harmful in the Linux kernel, multithread, etc. So, the technique in the ACCESS_ONCE macro (& friends) is a way to force an access with C volatile semantics only when it is needed, avoiding the disadvantages. Note that this was contentious enough to cause a GCC regression!
Answer: A cast to volatile on a "simple" variable is basically useless - it won't change the underlying access semantics, so this is not what you want.
Comment: if you cast anything to volatile, you'd probably expect to change the storage class of that anything to volatile.

It doesn't work like this, however.

Think of a cast as creating a temporary object of that wanted type (you wouldn't expect the underlying variable to change to a double if you'd cast an integer to double, would you?).
Comment: How do you know what he/she wants ?

Why Is ACCESS_ONCE so complex?


makefile run targets in parallel

Question: I've the following makefile, with two rules.

Is there a way to run these two rules in parallel, I mean to maximize
  the core's capabilities? I see this section but not sure that I got how to use it for my purpose, since I want to handle it within the makefile and not from the command line.

I.e. run module1 & 2 targets in parallel.
This is the makefile:
all: module1 module2

.PHONY: module1
module1:
    @echo "run module 1"
    DIR=$(PWD)
    @echo $(DIR)

.PHONY: module2
module2:
    @echo "run module2"

cleanup:
    fzr clean $(DIR)

Comment: What about `make -j2` is not working?
Comment: Probably OP is unfamiliar. Checkout the [man](https://www.gnu.org/software/make/manual/html_node/Parallel.html).
Comment: @Thomas - as I write in the question , I want to do it within the file and not with command, how can I do it ?
Comment: @kabanus - im familiar with this option as it written in the doc i've provided , but my question is a bit different , I want to somehow do it from the file
Comment: Is there a particular reason you cannot invoke `make` with the job flag?
Answer: You can set make options that you usually pass to make via its command line invokation in the makefile itself. Add this line to your makefile
MAKEFLAGS += -j2

and you can invoke make without the -j flag, it will still spawn two processes to build targets in parallel, when they aren't dependent on each other. To automatically determine the number of jobs to spawn, you can use this on linux
NPROCS = $(shell grep -c 'processor' /proc/cpuinfo)
MAKEFLAGS += -j$(NPROCS)

and on MacOS
NPROCS = $(shell sysctl hw.ncpu  | grep -o '[0-9]\+')
MAKEFLAGS += -j$(NPROCS)

Comment: @shopiaT `MAKEFLAGS += -j$(shell nproc || printf 1)` (`|| printf 1` so that if `nproc` isn't available you'll get a `1`).
Comment: Go with `@echo $(NPROCS)`.
Comment: I adjusted the answer.
Comment: This is pretty far afield, but w.r.t. @logicbloke, try `--output-sync=target` on GNU make 4+.
Comment: thanks but this is someting that I see in the docs, what I need it to do it within the file as I write in the question and not by command, how it can be done ?
Comment: thanks , 1+, is there way to determine how much process I can run ?because it's different for each machine ...
Comment: This works on a Mac `sysctl hw.ncpu`. On Linux try `lscpu`
Comment: `nproc` is better for this than `lscpu`; it will give ju just the number.
Comment: @Biffen - can you show how to use it with the flags in the makefile ? the `nproc`
Comment: @lubgr thanks a lot , ill try it now ,is it possible also in windows machine ?
Comment: @lugbr - just to verify in case I use the code you have provided and I use `echo` for MAKEFLAGS and I got `--jobserver-fds=3,4 -j` what does this numbers `3,4` is ?
Comment: Same behavior here. It seems to be an undocumented internal detail of the jobserver functionality, you can read about it [here](https://savannah.gnu.org/bugs/index.php?47392) or [here](http://make.mad-scientist.net/papers/jobserver-implementation/).
Comment: @lubgr - but what is actually mean, it wil run in 3 process ?
Comment: @lubgr - or 4 process , i've core i7 (2.2 GHz Intel Core i7 just as an example... )
Comment: No, it works as expected, the parameters following `--jobserver-fds` don't reflect the number of processes. I just checked it with similar `module3` and `module4` targets and manually changing the `NPROCS` variable.
Comment: @lubgr - so how can I know how much process I have to run the makefile ? I've tried with `@echo $(MAKEFLAGS)` should I do it in other way?
Comment: This is great, but the fact that it runs 3 or 4 in the same shell, that's not so fun. As the output becomes pretty much useless since it starts mashing up. Would be great if it was forking processes with separate outputs.
Answer: Even a MAKEFLAGS += -j$(NPROCS) would not work if you have '::' instead of ':' for your rule, as illustrated by that recent fix in Git 2.25.2 (March 2020)
See commit 2607d39 (18 Feb 2020) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 29b09c5, 02 Mar 2020)

doc-diff: use single-colon rule in rendering Makefile
Signed-off-by: Jeff King
When rendering the troff manpages to text via "man", we create an ad-hoc Makefile and feed it to "make".
  The purpose here is two-fold:

reuse results from a prior interrupted render of the same tree
use make's -j option to build in parallel

But the second part doesn't seem to work (at least with my version of GNU make, 4.2.1). It just runs one render at a time.
We use a double-colon "all" rule for each file, like:
all:: foo
foo:
  ...actual render recipe...
all:: bar
bar:
  ...actual render recipe...
...and so on...

And it's this double-colon that seems to inhibit the parallelism.
  We can just switch to a regular single-colon rule.
Even though we do have multiple rules for "all" here, we don't have any recipe to execute for "all" (we only care about triggering its dependencies), so the distinction is irrelevant.

makefile run targets in parallel


How to compute $\int_0^2(1+4t^2+9t^4)^{1/2}\text{d}t$?

Question: The original question was:
find the length $\ell$ of the curve $\gamma$ given the parametric equations:
$$x=t~~~~~ y=t^2~~~~~ z=t^3 $$
from $t=0$ to $t=2$
Comment: I can say by a simple look that this integral cannot be solved with elementary function. Indeed I bet that there will be some Elliptic Integral hidden inside the result.
Comment: yeah the wolfram solution involved nonelementary integrals. is there an easier way?
Comment: If this is your integral, then there are no easier ways. Unless you want a REALLY approximate solution, which would be really bad in the neighborhood of $2$. Are you sure about this? Seems a really strange exercise. Did you take it from a book?
Comment: It was a midterm question
Comment: It's also impossible to solve in easier ways. So I think of a book misprint, as you can see by the total result below.
Answer: I bet there must be some error in this question because the indefinite integral looks monstrous, as Kim Peek already forsaw. It must be kind of Elliptic one. (I calculated the integral with wolfram)

How ever numerically the value between $0$ and $2$ is $9.57057$ See here
Comment: Gosh, that was ways ways ways worse than what I thought! Surely it cannot be calculated by hands that easily (mostly in some basic calculus course). Thank you for the explicit result!
Answer: For sure, this is a monster and numerical integration would be required.
However, I was able to get "reasonable" results expanding the integrand as a Taylor series around $t=1$ and integrating between the given bounds. Since the integrand is now $$\sqrt{9 t^4+4 t^2+1}=\sum_{i=0}^n \alpha_i(t-1)^i+O\left((t-1)^{n+1}\right)$$ $$\int_0^2\sqrt{9 t^4+4 t^2+1}\,dt=\sum_{i=0}^n \alpha_i\int_0^2(t-1)^i\,dt=\sum_{i=0}^n \alpha_i \frac{1+(-1)^i}{1+i}=2\sum_{i=0}^n  \frac{\alpha_{2i}}{2i+1}$$
I give below a table for a few results
$$\left(
\begin{array}{cc}
n & integral \\
 0 & 7.48331 \\
 2 & 9.57050 \\
 4 & 9.59621 \\
 6 & 9.58476 \\
 8 & 9.56654 \\
 10 & 9.56044
\end{array}
\right)$$
which are very quickly "close" to the numerical value already reported in
loved.by.Jesus's answer.


Google Charts API not working on web2py application uploaded on pythonanywhere.com

Question: I have created a web2py application and uploaded it on pythonanywhere.com.
In my application, I am using Google Charts API to create graphs and charts.
It is working perfectly on my local system.
However, my application is unable to fetch this source when hosted on pythonanywhere.com and JavaScript console in the browser flashes the following error:
"google not defined"
Is there a way out to use Google Charts API on pythonanywhere.com?
<!DOCTYPE html>

        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge{{=not request.is_local and ',chrome=1' or ''}}">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>{{=response.title or request.application}}</title>
        <meta name="application-name" content="{{=request.application}}">
        <meta name="google-site-verification" content="">

    <input type="hidden" name="temp" id="temp" />
    <form action="">

                <select class="form-control" style="width: 200px" name="subjects_tab1" id="subjects_tab1" onchange="tab1_js(this.value);jQuery('#temp').val(document.getElementById('subjects_tab1').value);ajax('{{=URL('default','dd')}}',['temp'],'question');get_comments(this.value)">
            <option name="" value="" id="0"></option>
            {{for s in subjects:}}
            <option name="{{=s.id}}" id="{{=s.id}}" value="{{=s.id}}">{{=s.course_name}}</option>
            {{pass}}
            </select>

                    <select class="form-control" style="width: 900px" name="question" id="question" onchange="check(jQuery('#question').val())">
                    </select>

    </form>

Hi

    <input id="id1" name="id1" value="" type="hidden"/>
    <input type="hidden" id="err" name="err" ></input>

Comment: Please provide some code.
Answer: Google charts is all client-side, so there's nothing about PythonAnywhere that would prevent it from working. However, it's possible that your server-side code is behaving differently and returning values that your javascript isn't handling properly. Ensure that your ajax requests are getting the data they expect from the server.


Why HERE API reponds with empty "View" on valid url request?

Question: I'm trying to call HERE geocoder API, but on a valid url call the response body isn't fully initialized.
As shown below, I simply call a the API providing a valid url:
val url = UriComponentsBuilder.fromUriString(properties.geocodeUrl)
    .queryParam("app_id", properties.appId)
    .queryParam("app_code", properties.appCode)
    .queryParam("searchtext", "10 Downing Street, London")
    .toUriString()

logger.debug("calling geocoder rest service: $url")

val geocoderResponse = RestTemplate().getForObject<GeocoderResponse>(url)

GeocoderResponse
package com.gudjob.geolocation

import com.fasterxml.jackson.annotation.JsonProperty
import com.gudjob.core.domain.Coordinates
import java.io.Serializable

/**
 * @author Elia Sgolmin
 */
internal data class GeocoderResponse(
    @field:JsonProperty("Response")
    var response: Response?
) : Serializable {

    data class Response(
        @field:JsonProperty("View")
        var views: List<View>
    ) : Serializable {

        data class View(
            @field:JsonProperty("Result")
            var results: List<Result>
        ) : Serializable {

            data class Result(
                @field:JsonProperty("Location")
                var location: Location?
            ) : Serializable {

                data class Location(
                    @field:JsonProperty("DisplayPosition")
                    var displayPosition: DisplayPosition?
                ) : Serializable {

                    data class DisplayPosition(
                        @field:JsonProperty("Latitude")
                        var latitude: Double?,

                        @field:JsonProperty("Longitude")
                        var longitude: Double?
                    ) : Serializable {

                        fun toCoordinates(): Coordinates {
                            val lat = latitude
                            val lon = longitude
                            return if (lat == null || lat.isNaN() || lon == null || lon.isNaN())
                                Coordinates.Unavailable
                            else
                                Coordinates.Available(lat, lon)
                        }
                    }
                }
            }
        }
    }
}

If I copy the logged url and use it in browser or postman I get the following expected response
{
   "Response":{
      "MetaInfo":{
         "Timestamp":"2019-02-26T14:03:09.236+0000"
      },
      "View":[
         {
            "_type":"SearchResultsViewType",
            "ViewId":0,
            "Result":[
               {
                  "Relevance":1,
                  "MatchLevel":"houseNumber",
                  "MatchQuality":{
                     "City":1,
                     "Street":[
                        1
                     ],
                     "HouseNumber":1
                  },
                  "MatchType":"pointAddress",
                  "Location":{
                     "LocationId":"NT_lWsc8knsFwVitNTFX88zmA_xAD",
                     "LocationType":"point",
                     "DisplayPosition":{
                        "Latitude":51.50341,
                        "Longitude":-0.12765
                     },
                     "NavigationPosition":[
                        {
                           "Latitude":51.50322,
                           "Longitude":-0.12767
                        }
                     ],
                     "MapView":{
                        "TopLeft":{
                           "Latitude":51.5045342,
                           "Longitude":-0.129456
                        },
                        "BottomRight":{
                           "Latitude":51.5022858,
                           "Longitude":-0.125844
                        }
                     },
                     "Address":{
                        "Label":"10 Downing Street, London, SW1A 2AA, United Kingdom",
                        "Country":"GBR",
                        "State":"England",
                        "County":"London",
                        "City":"London",
                        "District":"Westminster",
                        "Street":"Downing Street",
                        "HouseNumber":"10",
                        "PostalCode":"SW1A 2AA",
                        "AdditionalData":[
                           {
                              "value":"United Kingdom",
                              "key":"CountryName"
                           },
                           {
                              "value":"England",
                              "key":"StateName"
                           },
                           {
                              "value":"London",
                              "key":"CountyName"
                           }
                        ]
                     }
                  }
               }
            ]
         }
      ]
   }
}

As you can see, "View" attribute of "Response" object is an array with one object containing latitude and longitude of the given address.
What I get instead is

GeocoderResponse(response = Response(views = []))

where response has been initialized, but views is empty.
Changing my code to get JSON response as plain text with
restTemplate().getForObject<String>(url)

it results in the following json

{"Response":{"MetaInfo":{"Timestamp":"2019-02-26T13:35:37.851+0000"},"View":[]}}

Even here "View" property is empty so I don't think the issue is about JSON deserialization.
Any suggestion?
UPDATE
I've checked RestTemplate request headers vs. browsers and postman ones, but couldn't find any relevant difference.
The only think I can point out is postman response header Content-Encoding → gzip, which is missing in Spring response.
Comment: Maybe response from server depends from [`Content Type`](https://stackoverflow.com/questions/477816/what-is-the-correct-json-content-type) or some other headers? Play with `postman` and try to find why it returns other response. Do you have an access to `REST` `API` method implementation you invoke?
Comment: There is no Content-Type since [Here's geocoder API](https://developer.here.com/documentation/geocoder/topics/what-is.html) uses url parameters.
The only header restTemplate adds is `Accept=[application/json, application/*+json]`. Giving this parameter to postman doen't change its (correct) response. About response headers, postman gets an additional one: `Content-Encoding → gzip` (and of course `Content
Comment: If you are sure everything is the same, you can do a `Unit Test` where you try to deserialise example response to object using `ObjectMapper`. It will prove that `POJO` structure fits `JSON` payload.
Comment: As I stated in my update, if I ask the response as a String, rest API responds with the quoted string and `View` is already empty, so I don't think it's a POJO problem
Comment: Try GSON, I tried it with GSON. It works perfectly.
Comment: @Moinkhan the issue is **not** about jackson vs. gson or others since even getting the body as a string I get empty `View`.
Answer: The issue might be ,the query parameter searchtext is not being URL encoded when requesting via the UriComponentsBuilder interface. Try encoding the string being passed and check the response.
.queryParam("searchtext", URLEncoder.encode("10 Downing Street, London","UTF-8"))


mod_rewrite with multiple query strings?

Question: I'm trying to cleanup some URLs on my blog, so I've decided to look into mod_rewrite. I haven't a clue what I'm doing though, so I was hoping I could get some help :P I have links like http://kn3rdmeister.com/blog/post.php?y=2012&m=07&d=04&id=4. Although it works, and people still get the content I want them to have, I don't like them having to look at all the query strings. I want to turn the above link into http://kn3rdmeister.com/blog/2012/07/04/4.php.
This is what my .htaccess looks like right now.
RewriteEngine On
RewriteCond %{QUERY_STRING} ^y=([0-9){4})&m=([0-9]{2})&d=([0-9]{2})&id=([0-9]*)$
RewriteRule ^/blog/post\.php$ http://kn3rdmeister.com/blog/%1/%2/%3/%4.php? [L]

Like I said, I'm absolutely clueless :D
Comment: It is much clearer if you only ask one question at a time. Please split one of them off into a separate post.
Comment: Regarding this question, what effect do these three lines in your .htaccess have on the example URL? Does it change at all?
Comment: Nothing changes. The .htaccess is in my website's root directory. Could it help if I make another .htaccess in the blog directory, and then edit the rule accordingly?
Comment: There is a typo in the pattern. The first ) should be a ]. Is that the post, or your actual .htaccess that has the typo?
Answer: If you're using apache 2.0 or higher, you're going to need to remove the leading slash (the prefix) if these rules are in an .htaccess file, so that your regular expression looks like this:
# also note this needs to be a "]"--v
RewriteCond %{QUERY_STRING} ^y=([0-9]{4})&m=([0-9]{2})&d=([0-9]{2})&id=([0-9]*)$
RewriteRule ^blog/post\.php$ http://kn3rdmeister.com/blog/%1/%2/%3/%4.php? [L]

This is going to make it so when someone puts http://kn3rdmeister.com/blog/post.php?y=2012&m=07&d=04&id=4 in their browser's URL address bar, their browser will get redirected to http://kn3rdmeister.com/blog/2012/07/04/4.php and the new URL will appear in their address bar.
I assume you've got something setup on your server to handle a request like blog/2012/07/04/4.php.
Comment: Thank you @Jon, that worked fine. However, I don't have anything setup to handle the redirection :P where would you suggest starting for that? this perhaps needs its own separate post.
Comment: You could probably use mod_proxy with ProxyPass and ProxyPassReverse. Might be overkill, but it should do the rewriting in both directions. https://httpd.apache.org/docs/2.0/mod/mod_proxy.html#proxypass
Comment: That looks promising... buuut I'm a n00b at this. I'll start a new question, as to keep things clean :P thanks for the help
Answer: At first you should define your URLs!!!
Like:
/blog shows front page
/blog/1234 shows post 1234
/blog/date/2012 shows posts by year
/blog/date/2012/06 shows posts by year and month
/blog/date/2012/06/01 shows posts by year and month and day
and so on...
First option is to rewrite each of your defined URLs to index.php. Your index.php has only to handle the submitted GET parameters.
### Do only if rewrite is installed
<IfModule mod_rewrite.c>

### Start rewrite and set basedir
RewriteEngine on
RewriteBase /

### Rewrite only if no file link or dir exists
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l

### Rewrite frontpage
RewriteRule ^blog$ /index.php?action=showfront [L,QSA]

### Rewrite post
RewriteRule ^blog/([0-9]+)$ /index.php?action=showpost_by_id&id=$1 [L,QSA]

### Rewrite posts by date
RewriteRule ^blog/date/([0-9]{4})$ /index.php?action=showposts_by_date&year=$1 [L,QSA]
RewriteRule ^blog/date/([0-9]{4})/([0-9]{2})$ /index.php?action=showposts_by_date&year=$1&month=$2 [L,QSA]
RewriteRule ^blog/date/([0-9]{4})/([0-9]{2})/([0-9]{2})$ /index.php?action=showposts_by_date&year=$1&month=$2&day=$3 [L,QSA]

### Rewrite posts by tag
RewriteRule ^blog/tag/([a-zA-Z0-9_-]+)$ /index.php?action=showposts_by_tag&tag=$1 [L,QSA]

</IfModule>

Test in index.php with:
    print_r($_GET);
    print_r($_POST);
The second option is to rewrite all URLs and your index.php needs to handle all possible URLs. So at first it needs something like a router that splits the incoming URL in parts and then send the requested page or an error-page. I would try this at first as the bloody school.
<IfModule mod_rewrite.c>

RewriteEngine on
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l

RewriteRule ^ index.php%{REQUEST_URI} [L]

</IfModule>

Test in index.php with:
print_r(explode('/', ltrim($_SERVER['PATH_INFO'], '/')));
print_r($_GET);
print_r($_POST);

The third option is to use a PHP framework. A framework may help you to write your code quite fast. It delivers you many base-classes like a router. (f.e. ZendFramework, Flow3, Kohana, Symfony, CodeIgniter, CakePHP, yii and others). This will make you more advanced.
The fourth and laziest option is to use a ready made software like Wordpress.
Comment: Woah... That's a lot of info to look at :D I'll look into all of that at a later time (a few hours, likely). I think I may need to reconsider how I display and post my content, so that I can expand the range of things I can play with.
Comment: Your "first option" answer turns /blog/date/2012/07/04/4.php into index.php?action=showposts_by_date&year=1234&month=05&day=21 . But the questioner wants the opposite effect.


Show that the adjusted values for 2x2 tables under the assumption of independence have the same absolute value

Question: I am trying to show that, for 2x2 tables, under the assumption of independence, that all four adjusted residuals have the same absolute value.
Adjusted residual: $r_{ij}=\dfrac{n_{ij}-\hat{u_{ij}}}{\sqrt{ \hat{u_{ij}}(1-P_{i+})(1-P_{+j})}}$
where $n_{ij}$ is the observed cell count, and $\hat{u_{ij}}=\dfrac{n_{i+}n_{+j}}{n}$ is the expected cell count, under the assumption of independence.
So far I have:
$r_{11}=\dfrac{n_{11}-\dfrac{n_{1+}n_{+1}}{n}}{\sqrt{ \dfrac{n_{1+}n_{+1}}{n}(1-\dfrac{n_{1+}}{n})(1-\dfrac{n_{+1}}{n})}}=\dfrac{n_{11}-\dfrac{n_{1+}n_{+1}}{n}}{\sqrt{ \dfrac{n_{1+}n_{+1}n_{2+}n_{+2}}{n}}}$
$r_{12}==\dfrac{n_{12}-\dfrac{n_{1+}n_{+2}}{n}}{\sqrt{ \dfrac{n_{1+}n_{+1}n_{2+}n_{+2}}{n}}}$
$r_{21}==\dfrac{n_{21}-\dfrac{n_{2+}n_{+1}}{n}}{\sqrt{ \dfrac{n_{1+}n_{+1}n_{2+}n_{+2}}{n}}}$
$r_{22}==\dfrac{n_{22}-\dfrac{n_{2+}n_{+2}}{n}}{\sqrt{ \dfrac{n_{1+}n_{+1}n_{2+}n_{+2}}{n}}}$
So I just need to show that the absolute values of the numerators are the same. I can show this is true if i pick some random values for the 2x2 table, but how do I show that it is true in general? Thanks
Comment: Please add the `[self-study]` tag & read its
[wiki](http://stats.stackexchange.com/tags/self-study/info).
Comment: All the denominators are the same.  Since this conclusion is correct, then algebra (along with the definitions of the $n_{1+}$ *etc.*) *must* show that all the numerators are the same up to sign.  Why not just do the algebra then?
Answer: Just some simplification might help. The denominators are all the same, so you might scratch those... and keep in mind that
$$n=n_{11}+n_{12}+n_{21}+n_{22}$$
Similarly, you could write $n$ as the sum of either the row totals or the column totals, for example,
$$n=n_{1+}+n_{2+}$$
Also, you can write each row total as a sum of the row entries, for example,
$$n_{1+} = n_{11}+n_{12}$$
Using these facts, it seems like your answer will fall out...


Increase google play store early access user limit

Question: I have uploaded an app and registered in the early access program. But when the number of users downloaded the app reached 100, no more users can download the app even though they are on the same G-Suite domain.
How can I solve this issue?
Comment: I'm voting to close this question as off-topic because it is not about programming.
Answer: All of the documents on this are here.
The purpose of early access is for testing and QA. If you need more than 100 users, you probably aren't just doing testing and QA. Your options are:

If you want to run a closed test and manage them by email address do a "Closed test". You can have up to 2,000 users
if you want anyone in the world to have early access run an "Open test"
if you want to limit it to a particular domain only use private apps


How to customize Android's LockScreen?

Question: I know that there is already some apps can do this, such as:
WidgetLocker Lockscreen

Flyscreen

I want to know how to do this in my app? What classes I can use? Or what articles I should read? I don't want to change the Android's source code.
And I wonder if it will be different to deal with when the screen is locked by draw pattern?
Comment: possible duplicate of [Developing a custom lock screen](http://stackoverflow.com/questions/5529608/developing-a-custom-lock-screen)
Answer: Have a look at this answer. You need to write your own home screen app which will implement the lockscreen behaviour that you require. There is sample code for writing your own home screen app in the Android SDK.

How to customize Android's LockScreen?