Majority Voting in perl?

Question: I have 5 files containing the same words. I want to read each word in all the files and decide the winning word by detecting the following characters in a word (*, #, $, &) separated by tabs. Then, I want to generate an output file. Ii can only have 2 winners. For example:
file1
    we$
    are*
    ...

file2
    we$
    are#
    ...

file3
    we&
    are*
    ...

file4
    we$
    are#
    ...

file5
    we$
    are&
    ...

output file:
we$
are*#

Here is how I started:
#!/usr/local/bin/perl -w

sub read_file_line {
  my $fh = shift;
  if ($fh and my $line = <$fh>) {
    chomp($line);
    return $line;
  }
  return;
}

open(my $f1, "words1.txt") or die "Can't";
open(my $f2, "words2.txt") or die "Can't";
open(my $f3, "words3.txt") or die "Can't";
open(my $f4, "words4.txt") or die "Can't";
open(my $f5, "words5.txt") or die "Can't";

my $r1 = read_file_line($f1);
my $r2 = read_file_line($f2);
my $r3 = read_file_line($f3);
my $r4 = read_file_line($f4);
my $r5 = read_file_line($f5);

while ($f5) {

    #What can I do here to decide and write the winning word in the output file?

$r1 = read_file_line($f1);
$r2 = read_file_line($f2);
$r3 = read_file_line($f3);
$r4 = read_file_line($f4);
$r5 = read_file_line($f5);
}

Comment: The smell, you know that [alcohol](http://wiki.xtronics.com/index.php/Shower_Board_as_a_white_Board#CAS_numbers) smell. The whole board. Smells like... homework.
Comment: Hey - the revised question is a trivial variant on the previous one; you should be able to adapt any of the previous solutions to the new scenario.  And changing the question so that the previous answers are all irrelevant is non-kosher.  So, too, is not marking the question as homework when it is clearly homework.  Then not thinking about your revised requirement is pure laziness on your part.
Comment: @aliocee: OK - you've learned.  Please remember for the future.  Thanks!
Comment: @aliocee: [homework on SO](http://meta.stackexchange.com/questions/10811/homework-on-stackoverflow).
Comment: I'm really sorry, i changed the question to make you understand what the second approach is not to make it irrelevant. its a project that i got stack into try to seek for your help. im sorry.
Answer: Sounds like the job for a hash of hashes. Untested code:
use strict;
use warnings;
use 5.010;
use autodie;
use List::Util qw( sum reduce );

my %totals;

my @files = map "words$_.txt", 1..5;

for my $file (@files) {
    open my $fh, '<', $file;
    while (<$fh>) {
        chomp;
        my ($word, $sign) = /(\w+)(\W)/;
        $totals{$word}{$sign}++;
    }
}

open my $totals_fh, '>', 'outfile.txt';

my @sorted_words = sort { sum values %{$totals{$a}} <=> sum values %{$totals{$b}} } keys %totals; #Probably something fancier here.

for my $word (@sorted_words[0, 1]) {
    #say {$totals_fh} $word, join('', keys %{$totals{$word}} ), "\t- ", function_to_decide_text($totals{$word});
    say {$totals_fh} $word, reduce {
            $totals{$word}{ substr $a, 0, 1 } == $totals{$word}{$b} ? $a . $b
          : $totals{$word}{ substr $a, 0, 1 } > $totals{$word}{$b} ? $a
          :                                                          $b;
    } keys %{ $totals{$word} };
}

EDIT: Forgot about the only two winners part. Fixed, somewhat.
EDIT2: Fixed as per comments.
Comment: You aren't storing the lines in a hash, just the count for each word and it's symbols; the data structure looks something like { are => { '$' => 10, '&' => 1 }, we => { '$' => 1, '#' => 11 } }; So chances are the hash will be nowhere near that big.
Comment: Well, just tested it, and it seems to runs just fine (generates the outfile with the two results and all) -- You'll have to define function_to_decide_text yourself, though.
Comment: If you didn't change the for my $word part, then the error should be in @sorted_words[0,1] - It's not getting filled. Try adding use Data::Dumper; say Dumper \@sorted_words; just before the for.
Comment: @aliocee, true enough. Just edited in a version that does what you want, although I'm positive that I did an awful job at it.
Comment: @aliocee: New requirements, uh. For this one to work, you'll need to change the regex to something like /(\p{Ll})\t(\p{Lu})/, and then change how the string concatenation in the reduce works. You'll have to figure that one out yourself, though.
Comment: If you don't say which error, then there's not much anyone can do.
Comment: Use of uninitialized value $word in hash element at maj.pl line 16, <$_[...]> line 1.
Use of uninitialized value $sign in hash element at maj.pl line 16, <$_[...]> line 1. for all 10 lines.
Use of uninitialized value $word in hash element at maj.pl line 16, <$_[...]> line 2.
Use of uninitialized value $sign in hash element at maj.pl line 16, <$_[...]> line 2.
Comment: Problem's in the regex. What are you using?
Comment: i have a question, what if the number of files increase from 5 to 10?
Comment: Change the map from 1..5 to 1..10; But for a better solution, if your files are all in the same directory, use File::Find::Rule; my @files = File::Find::Rule->file->name( qr!words[1-9][0-9]*\.txt$! )->in( directory_name_here ); If they aren't in the same directory, you could pass their paths as arguments to the script (then replace @files with @ARGV), or hardcode them.
Comment: how about the condition to determine the winner?
Comment: How did it change? I'd think you'd have to change the `$a . $b` to `$a . '|' . $b;` or `join('|', $a, $b);`. but that should be about it.
Comment: but now i have 10 files and 5 tags (from A,B,C,D to A,B,C,D,E) which means the decision have to change to determine the winner.
Comment: Why? It's still counting how many times any symbol appears, and still displaying the result(s) that show up the most.
Comment: /([a-z]+)\s+([A-Z]+)/; or /(\p{Ll})\t(\p{Lu})/; both are not working to match the word variable it give this error: Use of uninitialized value $word in hash element at maj.pl line 16, <$_[...]> line 1.
Comment: The second regex won't work because it's not meant to work: It's missing any qualifiers (+*?); at most, it'd match a lowercase letter, a tab, and an uppercase letter. You should seriously consider reading perlretut (http://perldoc.perl.org/perlretut.html). Dunno about the first though. Does it match your data in http://rubular.com/ or http://gskinner.com/RegExr/ ?
Comment: Then the data you posted earlier isn't in the same format as the one you are using. You'll have to toy around with the regex.
Comment: yes i told you i finished with the punctuation part, now im dealing with the letters part separated by tab(i.e. We   A, are   D, etc), yesterday i tried changing the question but Jonathan told me not to change because ur answers will be irrelevant.
Comment: I know. I've been telling you how to adapt this answer to those new requirements ever since. But you see, the data you posted then didn't have uppercase letters on the first part. See how that would affect all the regexes I've posted so far.
Answer: Test Data Generator
#!/usr/bin/env perl

use strict;
use warnings;

foreach my $i (1..5)
{
    my $file = "words$i.txt";
    open my $fh, '>', $file or die "Failed to open $file for writing ($!)";
    foreach my $w (qw (we are the people in charge and what we say goes))
    {
        my $suffix = substr('*#$&', rand(4), 1);
        print $fh "$w$suffix\n";
    }
}

Majority Voting Code
#!/usr/bin/env perl

use strict;
use warnings;

my @files = ( "words1.txt", "words2.txt", "words3.txt",
              "words4.txt", "words5.txt"
            );

my @fh;
{
    my $n = 0;
    foreach my $file (@files)
    {
        open my $f, '<', $file or die "Can't open $file for reading ($!)";
        $fh[$n++] = $f;
    }
}

while (my $r = process_line(@fh))
{
    print "$r\n";
}

sub process_line
{
    my(@fhlist) = @_;
    my %words = ();
    foreach my $fh (@fhlist)
    {
        my $line = <$fh>;
        return unless defined $line;
        chomp $line;
        $words{$line}++;
    }

    my $combo = '';
    foreach my $word (keys %words)
    {
        return $word    if ($words{$word} >  2);
        $combo .= $word if ($words{$word} == 2);
    }
    $combo =~ s/(\W)\w+(\W)/$1$2/;
    return $combo;
}

Example Data and Results
$ perl datagenerator.pl
$ perl majorityvoter.pl > results.txt
$ paste words?.txt results.txt
we*     we$     we&     we#     we#     we#
are*    are#    are#    are*    are$    are*#
the*    the&    the#    the#    the&    the&#
people& people& people$ people# people# people&#
in#     in*     in$     in*     in*     in*
charge* charge# charge& charge* charge# charge#*
and$    and*    and$    and&    and$    and$
what&   what&   what$   what&   what#   what&
we#     we*     we*     we&     we*     we*
say$    say&    say$    say$    say$    say$
goes$   goes&   goes#   goes#   goes#   goes#
$

This seems to be correct for the test data in the files generated.

Revised requirements - example output
The 'revised requirements' replaced the '*#$&' markers after the words with a tab and one of the letters 'ABCD'.  After some swift negotiation, the question is restored to its original form.  This output is from a suitably adapted version of the answer above - 3 code lines changed, 2 in the data generator, 1 in the majority voter.  Those changes are not shown - they are trivial.
we      C       we      D       we      C       we      C       we      D       we      C
are     C       are     D       are     C       are     B       are     A       are     C
the     B       the     D       the     A       the     A       the     D       the     A|D
people  D       people  B       people  A       people  B       people  D       people  B|D
in      D       in      B       in      C       in      B       in      D       in      D|B
charge  C       charge  D       charge  D       charge  D       charge  A       charge  D
and     A       and     B       and     C       and     C       and     B       and     B|C
what    B       what    B       what    B       what    C       what    C       what    B
we      D       we      B       we      D       we      B       we      A       we      B|D
say     D       say     D       say     B       say     D       say     D       say     D
goes    A       goes    C       goes    A       goes    C       goes    A       goes    A

Revised test generator - for configurable number of files
Now that the poster has worked out how to handle the revised scenario, this is the data generator code I used - with 5 tags (A-E).  Clearly, it would not take a huge amount of work to configure the number of tags on the command line.
#!/usr/bin/env perl

use strict;
use warnings;

my $fmax  = scalar(@ARGV) > 0 ? $ARGV[0] : 5;
my $tags  = 'ABCDE';
my $ntags = length($tags);
my $fmt   = sprintf "words$fmax-%%0%0dd.txt", length($fmax);

foreach my $fnum (1..$fmax)
{
    my $file = sprintf $fmt, $fnum;
    open my $fh, '>', $file or die "Failed to open $file for writing ($!)";
    foreach my $w (qw(We Are The People In Charge And What We Say Goes))
    {
        my $suffix = substr($tags, rand($ntags), 1);
        print $fh "$w\t$suffix\n";
    }
}

Revised Majority Voting Code - for arbitrary number of files
This code works with basically arbitrary numbers of files.  As noted in one of the (many) comments, it does not check that the word is the same in each file as required by the question; you could get quirky results if the words are not the same.
#!/usr/bin/env perl

use strict;
use warnings;

my @files = scalar @ARGV > 0 ? @ARGV :
            ( "words1.txt", "words2.txt", "words3.txt",
              "words4.txt", "words5.txt"
            );
my $voters = scalar(@files);

my @fh;
{
    my $n = 0;
    foreach my $file (@files)
    {
        open my $f, '<', $file or die "Can't open $file for reading ($!)";
        $fh[$n++] = $f;
    }
}

while (my $r = process_line(@fh))
{
    print "$r\n";
}

sub process_line
{
    my(@fhlist) = @_;
    my %words = ();
    foreach my $fh (@fhlist)
    {
        my $line = <$fh>;
        return unless defined $line;
        chomp $line;
        $words{$line}++;
    }
    return winner(%words);
}

# Get tag X from entry "word\tX".
sub get_tag_from_word
{
    my($word) = @_;
    return (split /\s/, $word)[1];
}

sub winner
{
    my(%words)   = @_;
    my $maxscore = 0;
    my $winscore = ($voters / 2) + 1;
    my $winner   = '';
    my $taglist  = '';
    foreach my $word (sort keys %words)
    {
        return "$word\t$words{$word}" if ($words{$word} >= $winscore);
        if ($words{$word} > $maxscore)
        {
            $winner = $word;
            $winner =~ s/\t.//;
            $taglist = get_tag_from_word($word);
            $maxscore = $words{$word};
        }
        elsif ($words{$word} == $maxscore)
        {
            my $newtag = get_tag_from_word($word);
            $taglist .= "|$newtag";
        }
    }
    return "$winner\t$taglist\t$maxscore";
}

One Example Run
After considerable experimentation on the data presentation, one particular set of data I generated gave the result:
We          A|B|C|D|E   2  B  C  C  E  D  A  D  A  E  B
Are         D           4  C  D  B  A  D  B  D  D  B  E
The         A           5  D  A  B  B  A  A  B  E  A  A
People      D           4  E  D  C  D  B  E  D  D  B  C
In          D           3  E  C  D  D  D  B  C  A  A  B
Charge      A|E         3  E  E  D  A  D  A  B  A  E  B
And         E           3  C  E  D  D  C  A  B  E  B  E
What        A           5  B  C  C  A  A  A  B  A  D  A
We          A           4  C  A  A  E  A  E  C  D  A  E
Say         A|D         4  A  C  A  A  D  E  D  A  D  D
Goes        A           3  D  B  A  C  C  A  A  E  E  B

The first column is the word; the second is the winning tag or tags; the third (numeric) column is the maximum score; the remaining 10 columns are the tags from the 10 data files.  As you can see, there two each of 'We A', 'We B', ... 'We E' in the first row.  I've also generated (but not preserved) one result set where the maximum score was 7.  Given enough repetition, these sorts of variations are findable.
Comment: @aliocee: THINK!  There are two lines to change in datagenerator.pl and one line to change in majorityvoter.pl - it is that trivial a change in requirements.  This is a site for LEARNING; that means you are expected to learn from the answers given you.
Comment: @aliocee: What I wrote was `$combo =~ s/\t([A-D])\w+\t([A-D])/\t$1|$2/;` which includes the tab character twice where yours doesn't bother.  Yours works fine; mine does too.  Nominally, mine might spot some erroneous data yours doesn't (missing tabs, specifically), but that's not really an issue here.
Comment: @aliocee: given N voters (files), what is the formula for when someone is the outright winner? (1 voter → 1 vote; 2 → 2; 3 → 2; 4 → 3; 5 → 3; ...).  If there isn't an outright winner, then (supposing you have 10 voters), you might see a score of, say 1, first, then 2, then later you might see a score of 3 (out of 10), and later again another score of 3.  Once you've seen a score of 2, all scores of 1 are uninteresting; once you've seen a score of 3, the scores of 2 are uninteresting. When scoring, you probably split the word and the tag (A-E) and store the word and the list of tags separately.
Comment: @aliocee: My previous comment gives you most of the algorithm.  You don't have to use the outright winner number.  You need to keep track of the current maximum number of votes.  When a new vote comes in with fewer votes, it is uninteresting and can be ignored.  When a new vote comes in with more votes, you discard the information about the previous high vote count and record the new high vote, the word, and the tag (A-E if there are 5 tags).  When the new vote has the same number of votes as the previous high vote count, you need to add its tag to the list of tags.  This gives you the winner.
Comment: I try editing your codes to implement the edited version of my question using (A,B,C,D instead of *,#,$,&) separated by tab character but i cant detect the majority word in the regular expression. any suggestion please? thanks.
Comment: @aliocee: and a Happy New Year to you, too!
Comment: yes, i put (A,B,C,D) into an array then randomise the index, i dont have any problem with the datagenerator.pl, its the majorityvoter.pl the is giving problem thats why i decided to ask you. this is what i change in the regular expression: $combo =~ s/(A|B|C|D)\w+(A|B|C|D)/$1$2/; but having problems.
Comment: @aliocee: the line that needs to be changed in the majority voter script is the substitute expression in the line reading `$combo =~ s/(\W)\w+(\W)/$1$2/;`.  You need to understand how that works in the original context; it is then trivial to adapt it to the revised requirement.  Use '[...]' character classes, and don't forget about the tabs you added.
Comment: $combo =~ s/(A|B|C|D)[\w+\t](A|B|C|D)/$1$2/; ?
Comment: @aliocee: more or less.  The combo contains a word, a tab, a letter from `[A-D]`, the word again, another tab, and another letter.  You need one of the words, a tab, and the two letters separated by a pipe.  In case of doubt, print intermediate values; that is standard debugging technique, especially for code that isn't your own that you don't fully understand (or even for code that you wrote but no longer fully understand!).
Comment: $combo =~ s/([A-D])\t\w+([A-D]\|[A-D])/$1$2/; here is what i understand from your explanation is this correct?
Comment: @aliocee: Your substitute looks for letter, tab, word, letter; where did the literal pipe and the next letter come from? Read my previous comment again, carefully. The substitute operation in the original bypassed the first word - started matching after the end of the word, at the punctuation character. So, your revised expression needs to start matching after the end of the first word, starting with a tab. Your replacement string will insert the pipe symbol; it isn't in `$combo` to be matched, as you'd see if you put a `printf "<<$combo>>\n";` statement before and after the substitute.
Comment: @aliocee: I'm not cross with you now - I probably was earlier.  I am trying to get you to think methodically, because that is the most crucial programming skill there is.  There is system in the way regexes work, and programs in general, and you need to learn how to see the system through the syntax.  That is, the syntax (especially for regexes) is obscure or ghastly (or both), but you have to see past the surface level of confusion to the method underneath.  See http://regex.info/ or http://www.regular-expressions.info/.
Comment: $combo =~ s/([A-D])\w+\t([A-D])/$1\|$2/; ?
Comment: @aliocee: Did that new version work for you?  It is marginally different from mine, but marginally neater too.  Fortunately, it's Perl and TMTOWTDI - there's more than one way to do it.  (That looks correct to me.)
Comment: for the record you are a get teacher, thanks alot for not getting angry with me sure u r a patient guy.
Comment: i verified its correct, now u know you that i can put effort to learn new things in programming. please can u tell me ur methed as well?
Comment: http://gskinner.com/RegExr/ You can check the validity of your regexen there; There's a bunch of sites like that, but this one includes support for recursive patterns and a bunch of cool stuff that hopefully you'll never have to use : ) For instance, to get my previous solution working, you'd need to change the regex to /(\p{Ll}+)\s+(\p{Lu})/.. Which isn't supported in the site I just linked you to. Dammit! You should take a look at perlretut, perlre, and this: http://www.perladvent.org/2010/1/
Comment: @Jonathan - Because my method will not match the word that start with capital letter from A-D, also i may change the array in the datagenerator.pl to be like @a = qw(AW BX CY DZ); instead of @a = qw(A B C D); base on my project. in this case, [A-D] expression will not work. but may be it will work on your method.
Comment: On the contrary! If your regex doesn't match the data, you can't change the data: Change the regex! /([a-z]+)\s+([A-Z]+)/ ought to do the trick.
Comment: About the last regex you posted.
Comment: @aliocee: Reading through your comment about initial capitals etc, I went and capitalized the sentence in the data generator, and then ran it through my voter script, and it worked fine because I explicitly looked for the tab, [A-D], word, tab, [A-D]; the leading tab protected me from all problems.  Maybe I should partially retract my previous comment - it is a good idea to look for that leading tab explicitly.  But it was reflexive on my part; I did it automatically without really noticing the repetition until you posted your variant.
Comment: @Jonathan - thanks alot for ur help, i just tested using what i explain to you in the last comment i sent to you, it is also working fine.
Comment: @Jonathan - in mine, its not working after i capitalised some of the words, which was why i asked for your solution.
Comment: @aliocee: As @Hugmeir noted, most people cannot simply change the data format to suit their needs - the data format is given and you have to adapt to what is given.  It is good that you are able to make it work with the alternative; it shows you are thinking about the problem and a possible solution.  However, be wary of changing the starting conditions unless you are supposed to be in charge of them.  (When you are in charge of the formats, you ensure that things are as easy to parse as possible; the punctuation suffix was an excellent example of 'easy to parse'.)
Comment: i have a question, what if the number of files increase from 5 to 10?
Comment: @aliocee: The array increases from 5 to 10; the conditions for which is the majority becomes more complex.  How much more so depends on whether the number of tags increases from 4 to some larger number...
Comment: Yes tags increases from 4 to 5 as well?
Comment: @aliocee: What rules have you come up with?  (1) When is there an outright winner? (2) How are you going to deal with the case where there are three tags all with 3 entries?  (3) How are you going to deal with the case where there are five tags all with 2 entries?  (4) Are there any other extrema to worry about?  You probably need to split the 'process_line()` sub into two - one part reads the line as now; the other deals with the majority voting stuff, which is about to get tougher.  Extending the array (from 5 to 10 to any number) is easy - that's the advantage of arrays.  Voting is harder.
Comment: @Jonathan - (1) a winner will be the one with higher number all the time it 1 or more. (2) all 3 will be winners separated with pipe character (3) all 5 will be winners as well separated with pipe character. (4) For now no extra this is the final stage.
Comment: im trying to figure out since yesterday, because you told me to try on my own but cant get the answer to the voting part, i cant sleep until i finish this last part.
Comment: @Jonathan - please can u guide me with a short algorithm?
Comment: sub process_line
{
    my(@fhlist) = @_;
    my %words = ();
    foreach my $fh (@fhlist)
    {
        my $line = <$fh>;
        return unless defined $line;
        chomp $line;
        $words{$line}++;
    }

    return voter(%words);

}

sub voter
{
   my(%words) = @_;

   foreach my $word (keys %words)
   {
        my @tags = split(/\t/, $word);

   }
}
Comment: @aliocee: the revised process_line() looks practically identical to mine - I called the second sub winner instead of voter, but that's about all. The outline of the voter sub is a start; you have a bit more work to do in the loop body. I have a constant $winscore; any word with at least that many votes is automatically the winner. I have variables $maxscore (initially zero), $winner and $taglist. When there's a new maximum score, I set things appropriately; when there's a new score with the same score, I add its tag to $taglist; otherwise, I ignore it. At the end, format up the answer. –
Comment: One might argue that at some point, the code should check that the word portion of each line from each file matches - so that one file does not say `hello\tA` while another says `goodbye\tZ` on the same line number.  However, that is nominally out of scope; the code shown above simply assumes that the word is the same.  It actually only matters if you have (say) two lines saying `hello\tA` and two other lines saying `goodbye\tB`; then you'll get told either `hello\tA|B` or `goodbye\tA|B` is the winner.  Note that the `$maxscore` algorithm would be correct with 26 tags (A-Z) and only 10 files.
Comment: All files have the same content, only the tags (A-E) differs from each other in the 10 files. therefore i can hello\tA|B as HELLO occur in all 10 files. you can easily understand from your datagenerator.pl file.
Comment: Yes, ten files are the maximum i have.
Comment: sub voter
{
   my(%words) = @_;

   my $max = 0;
   my ($win,$list) = ("","");

   foreach my $word (keys %words)
   {
        my @tags = split(/\t/, $word);

 if($words{$word} >  $max){
      $max = $words{$word};

             if ($list eq ""){
       $list .= $tags[1];
          $win = $tags[0];
      }
             else{
                $list .= "|" . $tags[1];
              $win = $tags[0];
             }
 }

   }

   return $win . "\t" . $list;
}
Comment: Here is my last version it seems working but i have to verify it. check it for me please.
Comment: @aliocee: Getting close, but not quite right.  If `$words{$word} > $max`, then you need to set `$max`, `$list`, and `$win` unconditionally to the new values; else if `$words{$word} == $max`, then you need to add the new tag to the list of tags; else you can ignore the new entry.  I generally aim to avoid the '.' operator, so I'd write: `return "$win\t$list";`, for example, or `$list .= "|$tags[1]";`.  I think it makes for clearer reading; I'm not sure about efficiency.
Comment: sub voter
{
   my(%words) = @_;

   my $max = 0;
   my ($win,$list) = ("","");

   foreach my $word (keys %words)
   {
        my @tags = split(/\t/, $word);

 if($words{$word} >  $max){

      $max  = $words{$word};
             $win  = $tags[0];
             $list = $tags[1];
 }
        elsif ($words{$word} >  $max){

             $list .= "|$tags[1]";
 }
        else{

 }

   }

   return "$win\t$list";
}
Comment: What a great teacher you are ! i really appreciate all what you've done for me.
Comment: @aliocee: the `elsif` condition needs `==` in place of `>`, but otherwise correct.  You normally wouldn't write `else {}` but it isn't technically wrong.
Comment: Please why did you set `$winscore;` to  constant ?
Comment: @aliocee: I set `$winscore` to `($voters / 2) + 1`; if I the new score is greater than or equal to `$winscore`, then this is the winning score with no possibility of any other score equalling or exceeding it - so the loop can terminate early.  It is a minor optimization.
Comment: @Jonathan - I have a question in my question list, the question is: Matching words in a sentence using Perl? please can you give me your suggestion. thanks
Comment: @aliocee: I suggest writing it up as a complete new question.
Comment: @JonathanLeffler - i have written a new question as you suggested, please take a look at it. the question is "How can i duplicate sentence base on words cooccurance? in perl". thanks
Answer: #!/usr/bin/perl

use strict;
use warnings;

my @files   = qw(file1 file2 file3 file4 file5);
my $symbols = '*#$&'; # no need to escape them as they'll be in a character class
my %words;

foreach my $file (@files) {
   open(my $fh, '<', $file) or die "Cannot open $file: $!";
   while (<$fh>) {
      if (/^(\w+[$symbols])$/) {
         $words{$1} ++; # count the occurrences of each word
      }
   }
   close $fh;
}

my $counter  = 0;
my $previous = -1;

foreach my $word (sort {$words{$b} <=> $words{$a}} keys %words) {

   # make sure you don't exit if two words at the top of the list
   # have the same number of occurrences
   if ($previous != $words{$word}) {
      last if $counter > 1;
   }
   $counter ++; # count the output
   $previous = $words{$word};

   print "$word occurred $words{$word} times.\n";
}

Worked when I tried it out...
Comment: Thanks, but i just want to create exactly the same file as the input files which contains only the winning words. but when i have 2 winners then i have to show both symbols. not count.
Comment: @aliocee Why not go with Hugmeir's answer then? It seems rather elegant! And in case it doesn't quite meet your needs - just modify it a bit...

Majority Voting in perl?


C function pow() won't execute in shared hosting server

Question: I'm trying to run a C program through PHP exec() function. All is running well in my localhost machine, but when running it in hosting server, C program with pow() function that have variables as argument just won't execute.
I asked the customer support of that hosting, they said that executing C programming language is not supported on their server.
But what is strange for me is, why is my another C program is running well on that server, it just that one code that has pow() with variable as parameter in it that won't run.
Here is my code:
PHP
public function index() {
  //runs well
  $varExec = exec(app_path() . "/Exec/checkfuncpow.exe");
  var_dump($varExec);
  echo "
";
  //this the one that is not running
  $varExec = exec(app_path() . "/Exec/checkfuncpowwithvar.exe");
  var_dump($varExec);
}

Results:
string(8) "9.000000"
string(0) ""
checkfuncpow.c
#include <stdio.h>
#include <math.h>

int main() {
  printf("%f", pow((double)3, (double)2));
  return 0;
}

checkfuncpowwithvar.c
#include <stdio.h>
#include <math.h>

int main() {
  double param = 3;
  printf("%f", pow(param, (double)2));
  return 0;
}

Both code was compiled with the mandatory -lm flag to link to math library
gcc checkfuncpow.c -o ./Exec/checkfuncpow.exe -lm
gcc checkfuncpowwithvar.c -o ./Exec/checkfuncpowwithvar.exe -lm

So the ultimate question is is that really because the server does not support it? Or it should be able to run but the customer support didn't have a clue about the problem , so they just saying what is easy for them?
Comment: What OS and version are you running?  What OS and version is your hosting server running?
Comment: By "not supported" they don't mean it will not work, they mean they will not help you to make it work and it may break by any change they make and they don't invest time to fix it. It can be that it just happens to work for some reason but you shouldn't rely on that.
Comment: My guess is that there is no C math library. Maybe you could use dlopen(), dlsym(), (or LoadLibraryA() if you are on Windows) ... when it is available, to check for the c math library? You could detect if this is actually the problem. Otherwise you could statically link the library.
Comment: First of all i dont know this shared hosting operating system, could be linux. My operating system (localhost) is linux ubuntu 20.04. In the shared hosting where should i run these dlopen(), dlsym() functions to check math library?
Answer: The canonical answer: if your hosting service doesn't support executing C programs, then you should not do it; first of all, because programs may have bugs or strange behavior (like in this case), and second, because the hosting service would not give you any support in case of problems (like in this case).
The interesting answer: the mystery that is going through here is probably related to two things: 1) compiler optimization; 2) missing math library.
About compiler optimization, your two programs happen to behave differently because in the first case (no param variable), the compiler replaces pow((double)3, (double)2) with (double) 9.0; therefore there is no call to pow(), and a math library is not necessary. In the second case, the compiler doesn't optimize the pow() call away, so a math library is still necessary. If my guess is correct, and you compile both your programs with gcc option -O3, both programs should now run without problems.
About missing math library: the math library is linked to your program dynamically (with -lm option); that means that the target platform where the program will run needs to have the library installed. On the other hand, statically-linked libraries get embedded in your program so that the target platform is not required to have them installed.
One thing you could try is to statically link the math library (libm). Search about the topic if you don't know how to do it.
EDIT: in order to solve the problem statically-linking libraries into your executable, you can have a look at this StackOverflow question. TL;DR: GCC option -static should tell the compiler to use static libraries instead of dynamic ones, as long as static version of the library is available. In the case of libm, on my Ubuntu 20.04 box, static version of libm is available, so if I compile a program with command gcc -o program main.c -lm -static, GCC successfully use static version of libm.
Comment: I edited my answer showing you how to solve your problem statically-linking `libm` library. TL;DR: use command `gcc -o program main.c -lm -static` in order to compile your executable.
Comment: It's strange because libm is usually incorporated into libc these days (at least on Linux), even though the compiler and linker pretend it isn't.
Comment: It's also strange that the program runs at all if it has a dynamic dependency on `libm.so` and there's no `libm.so`. And if it is there, why the strange output?
Comment: I would not make any guess since the target platform appears to be Windows (in the question, executables have extension `.exe`). The GCC version being used here is probably taken from [winlibs.com](https://winlibs.com), which uses MinGW-w64 C library. I have not enough experience with GCC on Windows, so I don't know the cause of this strange behavior.
Comment: thank you for the reply i will definetly try -03 gcc option. and about libm im still confuse, could anyone point me to right direction. Im not using windows, i compile using linux ubuntu. .exe extension its just my habit of compiling C in .exe.
Comment: @LucaPolito Thank you so much, that really solve the problem, now it runs perfectly in the hosting server.


Enable nginx browser caching for specific url

Question: How can I enable nginx cache for specific URL? I have a django web app and I put my static files such as CSS, JavaScript and ... in a folder called static and I can access them via http://example.com/static/ and I want to config nginx n order to catch these files only from this url. I've read http://www.nginxtips.com/how-to-enable-browser-cache-static-files-on-nginx/ but this option wasn't explained there. I tried below config and restarted my nginx server but I've got errors.
This is my  sites-available/default config file. What's the problem?
server {
        listen  80;
        listen  443 ssl;
        server_name     www.example.com example.com;
        ssl_certificate /var/www/example/ssl/ssl.crt;
        ssl_certificate_key     /var/www/example/ssl/ssl.key;
        location /static (jpg|jpeg|png|gif|ico|css|js)$ {
                alias   /var/www/example/static;
                expires 3d;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";

        }
        location /media  {
                alias  /var/www/example/media;
        }
        location / {
                proxy_pass      http://127.0.0.1:8000;
        }
}

And this is nginx log file content...
2014/03/07 13:02:28 [error] 25189#0: *22 open() "/etc/nginx/html/static/css/wait.css" failed (2: No such file or directory), client: 188.8.131.52, server: www.example.com, request: "GET /static/css/wait.css HTTP/1.1", host: "example.com", referrer: "http://example.com/"
2014/03/07 13:02:28 [error] 25189#0: *23 open() "/etc/nginx/html/static/img/infographi.png" failed (2: No such file or directory), client: 18.104.22.168, server: www.example.com, request: "GET /static/img/infographi.png HTTP/1.1", host: "example.com", referrer: "http://example.com/"
2014/03/07 13:02:29 [error] 25189#0: *23 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 126.96.36.199, server: www.example.com, request: "GET /favicon.ico HTTP/1.1", host: "example.com"
2014/03/07 13:08:41 [emerg] 25220#0: invalid location modifier "/static" in /etc/nginx/sites-enabled/default:8
2014/03/07 13:08:41 [emerg] 25221#0: invalid location modifier "/static" in /etc/nginx/sites-enabled/default:8
2014/03/07 13:09:50 [emerg] 25255#0: invalid location modifier "/static" in /etc/nginx/sites-enabled/default:8
2014/03/07 13:09:50 [emerg] 25256#0: invalid location modifier "/static" in /etc/nginx/sites-enabled/default:8

Answer: From the nginx wiki it seems you have a space in between "/static" and your regex pattern. Location takes only one "uri" argument, not two...and nginx will interpret that space as an argument delimiter.
Also you need to tell a location block you want to match a regular expression with the '~' or '~*' modifiers.
You might try this
location ~* "/static.*(jpg|jpeg|png|gif|ico|css|js)$" {

instead of what you have and see if that does what you want.
Answer: Do you have any other files in this folder? I you don't or if it's ok to have them exposed you could simplify config:
server {
        listen  80;
        listen  443 ssl;
        server_name     www.example.com example.com;
        ssl_certificate /var/www/example/ssl/ssl.crt;
        ssl_certificate_key     /var/www/example/ssl/ssl.key;
        root /var/www/example;

        location / {
                proxy_pass      http://127.0.0.1:8000;
        }

        location /media/ {
                # it's ok to have empty location
        }

        location /static/ {
                expires 3d;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
}


What is a proof of Euler's formula $e^{ix}=\cos(x)+i \sin(x)$ that doesn't use differentiation or De Moivre's theorem, if it exists at all?

Question: The Taylor series expansions of $e^x$, $\sin(x)$ and $\cos(x)$ are the standard way to prove it, but does there exist an elementary way that doesn't use limits, calculus or De Moivre's theorem?
Comment: What is your definition of $e^{ix}$, $\sin(x)$, $\cos(x)$ that does not use Taylor series or limits?
Comment: If you don't understand calculus or De Moivre's theorem, take the time to understand them as they build up your understanding of mathematics. It might be tempting to run before you can walk, but there really is no way to fully understand Euler's formula before you dive into the required mathematics.
Comment: You need to clarify if $x$ is real or complex number. Please also specify what you mean by those symbols $e^{ix}, \cos x, \sin x$ for real or complex $x$. Usually textbooks do not provide their clear meaning when these symbols are introduced to students and are shameless in not being explicit about it. Assumption is that there is no need to know the meaning for beginners.
Answer: The number $e$ is transcendental, so it seems impossible to obtain any non-trivial result about it that does not involve infinite processes (limits, derivatives, integrals...). If you want to define the exponential function for irrational arguments using the basic operations, it indeed seems difficult to avoid polynomials or rational functions, and these must be of infinite degree. Finally, to establish the theorem, you need some definition of the trigonometric functions that is not based on the complex exponential. For this you need to resort to well-axiomatized geometry and to establish the equivalence with statements in calculus. In the end, it boils down to computing the arc length along a circle, hence establish $\pi$, another transcendental number.
Answer: If you take the trigonometric addition formulas for granted,
$$(\cos(n)+i\sin(n))(\cos(1)+i\sin(1))
\\=\cos(n)\cos(1)-\sin(n)\sin(1)+i(\cos(n)\sin(1)+\sin(n)\cos(1))
\\=\cos(n+1)+i\sin(n+1)$$
and by induction,
$$\cos(n)+i\sin(n)=(\cos(1)+i\sin(1))^n.$$
From this we have
$$\left(\cos\left(\frac nm\right)+i\sin\left(\frac nm\right)\right)^m=\cos(n)+i\sin(n)=(\cos(1)+i\sin(1))^n$$ and
$$\cos\left(\frac nm\right)+i\sin\left(\frac nm\right)=(\cos(1)+i\sin(1))^{n/m}$$ where we extract some to some $m^{\text{th}}$ root. In other words, with $q$ a rational number,
$$\cos(q)+i\sin(q)=(\cos(1)+i\sin(1))^q.$$
If we denote the complex number $\cos(1)+i\sin(1)$ as $e^i$, we have
$$\cos(q)+i\sin(q)=(e^i)^q.$$

This is by no means a proof (and if you wanted to generalize to the reals, you would need limits anyway), but it shows the connection between the addition formulas, which can be proven by geometry, and the powers of a complex.

Disclaimer: I didn't say that $(e^i)^q=e^{iq}$.
Answer: I think we need to go in the opposite direction to get an understanding of why to choose (yes we actually get to choose) to expand the real exponential function to have the property that $$e^{i\phi} = \cos(\phi) + i\sin(\phi)$$
In the analysis of one complex variable there is introduced the concept of analytic continuation. As far as I remember this was the first good explanation of why to choose to expand the exponential function in this way.
If we do not put any particular restrictions on how well behaved a function shall be when we expand it's domain, then a lot of weird things can happen. The degrees of freedom of how to expand functions is basically limitless. There exist some really beautifully and disturbingly complicated functions in the wild.
But if we want to preserve well behavedness in the sense of complex analyticity then we will need to choose the one which obeys De Moivre.

If we disregard differentiation altogether, then we suddenly have no particular reason to define the real valued exponential function to be smooth at all, and there's no reason as to why the number $e$ would be special. Remember what makes the number $e$ special is that $$\mathfrak D(e^{x}) = e^x$$
We could use base two instead and expand from integers to reals with a floor function or something. Without differentiability, why should our function be smooth?
Answer: The relationship between $e^x$ and $\sin(x)$ and $\cos(x)$
You know know the the $\text{exp()}$ function is a special one, and has many interesting properties among them is that expression $ e^{x} = \lim_{n \to \infty} (1+\frac{x}{n})^n $
If you expand this with binomial theorem, and put $n \rightarrow \infty$, you would get the series for $\text{exp()}$
$$ e^{x} = \lim_{n \to \infty} (1+\frac{x}{n})^n $$
$$ e^{x} = 1+x+\frac{x^2}{2}+\frac{x^3}{6}+\frac{x^4}{24}+\frac{x^5}{120}+\frac{x^6}{720}+\frac{x^7}{5040} + \dots \dots $$
$$ e^{x}= \sum_{k=0}^{\infty}\frac{x^k}{k!}$$
From this series, replace $x$ with $xi$, where $i = \sqrt{-1}$ and split the result into real and imaginary section
$$ e^{x} = 1+x+\frac{x^2}{2}+\frac{x^3}{6}+\frac{x^4}{24}+\frac{x^5}{120}+\frac{x^6}{720}+\frac{x^7}{5040} + \dots \dots $$
$$ e^{xi} = 1+xi+\frac{(xi)^2}{2}+\frac{(xi)^3}{6}+\frac{(xi)^4}{24}+\frac{(xi)^5}{120}+\frac{(xi)^6}{720}+\frac{(xi)^7}{5040} + \dots \dots $$
$$ e^{xi} = \{ 1-\frac{x^2}{2}+\frac{x^4}{24}-\frac{x^6}{720}+ \dots \} + \{ x-\frac{x^3}{6}+\frac{x^5}{120}-\frac{x^7}{5040} + \dots \} \cdot i$$
$$\sin(x) = x-\frac{x^3}{6}+\frac{x^5}{120}-\frac{x^7}{5040} + \dots $$
$$\cos(x) = 1-\frac{x^2}{2}+\frac{x^4}{24}-\frac{x^6}{720}+ \dots $$
Therefore $e^{xi} = \cos(x)+i\sin(x)$


Random biased coins problem

Question: So the problem goes something like this:
There are 3 coins in a bag (C1, C2, C3).  C1 comes up heads $1/3$ of the time, C2 $1/2$ of the time, and C3 $3/4$ of the time.  Now you choose a coin at random, flip it, record the outcome and without replacing the coin back in the bag, pick another coin (out of the 2 remaining) and flip it.
a) What is the probability that 1 heads and 1 tails is flipped and b) what is the probability that a heads was flipped on the 1st flip given a tails was flipped on the 2nd.
I have figured out how to go about this if we put the coin back into the bag.  In that case the 1 heads and 1 tails is $P(H_1T_2) + P(T_1H_2)$ which using a probability tree I found to be 95/216 (not sure if this is correct, just what I got as an answer).  Same way for b) i found the answer to be 228/432 when you put the coin back into the bag.
I am not sure how to go about finding the solution to when the coin is not put back into the bag.  Any help would be appreciated.
I am fairly new to probability so if the problem isn't well stated I apologize.
Answer: You can still make a tree for the version without replacement. Here's a large part of a rough sketch of one:

The first three-way branching represents the first draw; the two-way branching that follows represents the result of the first coin flip, heads or tails; the two-branching below that represents the second draw; and the final two-way branching, which I didn't draw in its entirety, represents the result of the second coin flip. The probabilities for the coin flips depend on which coin is being flipped; the probabilities for the second draw are $1/2$ for each of the coins remaining in the bag.
To solve the first part of the problem using this method, just complete the tree and add up the probabilities of the final states corresponding to getting a head and a tail. For the second part, add up the probabilities of the final states that had a tail on the second flip; call that total $p_2$. Then add up the probabilities of the final states that had tails on both flips; call that total $p_{12}$. The probability that the first flip was tails given that the second was tails is then $\dfrac{p_{12}}{p_2}$.
Comment: This is also very helpful, thank you.  Also just a follow up question which may not be appropriate for this forum but how did you make that tree diagram, I have been trying to learn latex and cannot figure out how to make a tree diagram on it.
Comment: @Lok: I did a quick and dirty drawing in Paint, actually, and uploaded it into my answer.
Answer: For a) you need to work out all the cases.  One way you can get 1H+1T is to pull C1 first, get a head, then get a tail.  The chance of this is $\frac 13$(pull C1)$\frac 13$(get head)$[\frac 12\cdot \frac 12$(pull C2 and get tail)$+\frac 12 \cdot \frac 14$(pull C3 and get tail)$]=\frac 19(\frac 14 + \frac 18)=\frac 1{24}$.  If you are careful, all the possibilities will be disjoint and you can add them.  Not replacing is represented by the fact that the second draw is restricted to C2 and C3 (given that you start with C1).
For b) you need to calculate the probability $\frac{H_1T_2}{H_1T_2+T_1T_2}$.  $H_1T_2$ is just part of the result of a).  $T_1T_2$ is calculated similarly.
Comment: Ok, this helps a ton, thank you.

Random biased coins problem


Check value of next line without moving streamreaders pointer

Question: I got a .csv file which looks like this:

I am using C# Streamreader to read every line.
I want to do following:
When my Streamreader is at the first line I want to check the value of the second line without moving my Streamreaders pointer to the second line.
I am looking for something like Streamreader.Peek()
But Peek() only checks if there's ANYTHING in the next line or not.
I want to check the value of the next line and decide what to do (without moving the Streamreaders pointer).
Comment: That definitely sounds like an XY problem. What is it that you're actually trying to do? In order to read the data of the next line, you have to advance the reader to the next line, it's as simple as that. Why don't you want to advance the reader? Do you want to rewind it instead? Or don't you rather want to save the current line in memory and then read the next?
Comment: It reads a stream.  The nature of streams is that they come by and never come by again.  If you want to read and process things one line ahead, create a variable called `lastLine` and keep track of the current and last line.
Comment: Is reading all lines into an array and then parsing that array a viable option?
Answer: You can make new stream based on decorator pattern (for one line buffer)


Mixing with Maschine 2 software

Question: I have recorded a multi group sequence in Maschine 2.
In the mixing mode, if I am on a group that is a drum kit, I am able to adjust the volumes of each individual track without issue.
However, on the group that I recorded all the other sounds, whenever I try to adjust the volume of a specific track, all of the sliders move.
What I have noticed is that on the group for the drum kit, when i click on the sound heading, the whole area highlights, when I am on the group with my other recorded sounds, clicking on the sound heading does not highlight the whole area.
What am I missing here, how do I switch it so that I can adjust the individual track volumes from the mixer?
Below I have added a coupe of screen shots to show how on the drum kit group the whole track highlights, while on the other, it doesn't highlight.
Drum Kit Group:

Main Sounds Group:
Comment: You will probably get a better response if you post this in the NI forums, if you get an answer post it in here too.
Answer: Hold [shift] and click any track.
The reason this is happening is because you have all of the tracks in the group highlighted/selected. Just hold [shift] + click any track in the mixer view -or- in the group view.

Mixing with Maschine 2 software


Arranged in ascending order excluding one item

Question: I have this table

Here I want to arranged table in ascending order order by firstname and those who are from UK by customer id. I tried doing this
select *
from tblCustomers
order by FirstName, LastName, age asc
where not country = 'uk'

but we can't do it as where clause should be used after table name and when we do
select *
from tblCustomers
where not country = 'uk'
order by FirstName, LastName, age asc

It exclude UK which is not my desired solution. How can I arranged in ascending order, order by firstname and firstname from UK from customer Id?
Comment: As per the question guide, please do not post images of code, data, error messages, etc. - copy or type the text into the question. Please reserve the use of images for diagrams or demonstrating rendering bugs, things that are impossible to describe accurately via text.
Comment: Use `CASE` expression? As mentioned, your question doesn't make much sense, and you seem to understand what the `WHERE` does; it filters the data that would be returned, it has nothing to do with the order of the data returned.
Comment: [Conditional ORDER BY](https://sqlperformance.com/2012/08/t-sql-queries/conditional-order-by)
Comment: Its not clear what you desired results are... how do you want to order the customer from the UK?
Comment: And you say "excluding one item" but there are 2 from the UK. And then you say "It exclude UK which is not my desired solution" so you do want to exclude them and you don't want to exclude them? Very hard to know what you are looking for.
Comment: @DaleK I want to order uk one by customer Id
Comment: You realise that storing age is a bad thing to do? It changes every year...
Comment: And the columns you show in your image are different to the columns in your query.
Answer: Build the order by clause up using case expressions e.g.
declare @tblCustomers table (customer_id int, first_name varchar(12), last_name varchar(12), age int, country varchar(3));

insert into @tblCustomers (customer_id, first_name, last_name, age, country)
values
(1, 'John', 'Doe', 31, 'USA'),
(2, 'Robert', 'Luna', 22, 'USA'),
(3, 'David', 'Robinson', 22, 'UK'),
(4, 'John', 'Reinhardt', 25, 'UK'),
(5, 'Betty', 'Doe', 28, 'UAE');

select *
from @tblCustomers
order by
    case when country = 'UK' then 1 else 0 end asc -- non-UK first
    , case when country = 'UK' then customer_id else null end asc -- UK ordered by id
    , first_name, last_name, age asc; -- Non-UK ordered as specified

Returns:

customer_id
first_name
last_name
age
country

5
Betty
Doe
28
UAE

1
John
Doe
31
USA

2
Robert
Luna
22
USA

3
David
Robinson
22
UK

4
John
Reinhardt
25
UK

Notes:

please don't use images as we can't copy and paste the data
ideally provide the DDL+DML (as I have shown) as this makes it much easier to answer
always ensure your question is consistent, with the same table and columns names throughout
always provide your desired results
don't store age in a database, store date of birth and calculate age.


Basketball Betting Pool - Expected Number of Points...

Question: There are 64 teams who play single elimination tournament, hence 6 rounds, and you have to predict all the winners in all 63 games. Your score is then computed as follows: 32 points for correctly predicting the final winner, 16 points for each correct finalist, and so on, down to 1 point for every correctly predicted winner for the first round. (The maximum number of points you can get is thus 192.) Knowing nothing about any team, you flip fair coins to decide every one of your 63 bets. Compute the expected number of points.
I'm not exactly sure how to do this...
I assume everything is independent...
Working backwards, in the final, there is 1 game and a 50% chance of winning, in the semi-final, there are 2 games each with a 50% chance of winning, etc.
$$E[Points]= (1/2)\cdot 32 + (1/2)\cdot (1/2)\cdot 16 +$$
Is this the right approach?
Thanks.
Comment: Not sure this is clear.  If this is done as in the March Madness bracket, then you are required to pick all the winners in advance, before a single game is played.  In that case, of course, the probability that the final game is a $\frac 12$ matter is contingent on your chosen winner actually making it that far (a low probability event).
Answer: I am assuming you have to pick the winners of every game before the tournament starts.  Hint:  For a particular second round game there are four possible winners, so your chance of picking the correct winner is $\frac 14$.  There are $16$ second round games, so your expected score from the second round is ???  Follow this logic for each round.
Comment: Oh...I see...Thanks!


How to select rows from matrix with unique entry in specific column?

Question: I tried to solve this using a functional way, but I am not having much success.
Suppose there is a list of lists, and it is required to pick only the lists in it which have unique entry in specific position.
For example, suppose there is a matrix, and we want to select only the rows which have unique elements in the first column.
Here is an example:
INPUT:
list= {{ 1,2}, {1,3},{4,5}}

I'd like the output to be
list={{1,2},{4,5}}

It does not matter which 'row' to remove, the first one is ok, but any one is fine.
I tried Select, DeleteCases, DeleteDuplicates, Union, and few other things, but can't get it to work. I do not know how to tell Mathematica to only look for 'unique' element. Union comes close but it works on complete list. i.e. I do not know what to write for criteria, as in
DeleteDuplicates[list, <now what?> ]

For reference, this is how I do the above in Matlab:
EDU>> A=[1 2;1 3;4 5]

A =
     1     2
     1     3
     4     5

EDU>> [B,I,J]=unique(A(:,1));
EDU>> A(I,:)

ans =
     1     3
     4     5

thanks
Answer: Here is one way:
DeleteDuplicates[list, First@#1 === First@#2 &]

EDIT
Note that the timings and discussion below are based on M7
Upon reflecting a bit, I found a solution which will be (at least) order of magnitude faster for large lists, and sometimes two orders of magnitude faster, for this particular case (probably, a better way to put it is that the solution below will have different computational complexity):
Clear[delDupBy];
delDupBy[nested_List, n_Integer] :=
  Module[{parts = nested[[All, n]], ord, unpos},
    ord = Ordering[parts];
    unpos = Most@Accumulate@Prepend[Map[Length, Split@parts[[ord]]], 1];
    nested[[Sort@ord[[unpos]]]]];

Benchmarks:
In[406]:=
largeList = RandomInteger[{1,15},{50000,2}];

In[407]:= delDupBy[largeList,1]//Timing
Out[407]= {0.016,{{13,4},{12,1},{1,6},{6,13},{10,12},{7,15},{8,14},
            {14,4},{4,1},{11,9},{5,11},{15,4},{2,7},{3,2},{9,12}}}

In[408]:= DeleteDuplicates[largeList,First@#1===First@#2&]//Timing
Out[408]= {1.265,{{13,4},{12,1},{1,6},{6,13},{10,12},{7,15},{8,14},{14,4},
      {4,1},{11,9},{5,11},{15,4},{2,7},{3,2},{9,12}}}

This is particularly remarkable because DeleteDuplicates is a built-in function. I can make a blind guess that DeleteDuplicates with user-defined test is using a quadratic-time pairwise comparison algorithm, while delDupBy is n*log n in the size of the list.
I think this is an important lesson: one should pay attention to built-in functions such as Union, Sort, DeleteDuplicates etc when using custom tests. I discussed it more extensively in  this Mathgroup thread, where there are also other insightful replies.
Finally, let me mention that exactly this question has been asked (with the emphasis on efficiency) before here. I will reproduce here a solution I gave for the case when the first (or, generally, n-th) elements are positive integers (generalizing to arbitrary integers is straightforward).:
Clear[sparseArrayElements];
sparseArrayElements[HoldPattern[SparseArray[u___]]] := {u}[[4, 3]]

Clear[deleteDuplicatesBy];
Options[deleteDuplicatesBy] = {Ordered -> True, Threshold -> 1000000};
deleteDuplicatesBy[data_List, n_Integer, opts___?OptionQ] :=
  Module[{fdata = data[[All, n]], parr,
  rlen = Range[Length[data], 1, -1],
  preserveOrder =  Ordered /. Flatten[{opts}] /. Options[deleteDuplicatesBy],
  threshold =  Threshold /. Flatten[{opts}] /. Options[deleteDuplicatesBy], dim},
  dim = Max[fdata];
  parr = If[dim < threshold, Table[0, {dim}], SparseArray[{}, dim, 0]];
  parr[[fdata[[rlen]]]] = rlen;
  parr = sparseArrayElements@If[dim < threshold, SparseArray@parr, parr];
  data[[If[preserveOrder, Sort@parr, parr]]]
];

The way this works is to use the first (or, generally, n-th) elements as positions in some
huge table we preallocate, exploiting that they are positive integers). This one can give us crazy performance in some cases. Observe:
In[423]:= hugeList = RandomInteger[{1,1000},{500000,2}];

In[424]:= delDupBy[hugeList,1]//Short//Timing
Out[424]= {0.219,{{153,549},{887,328},{731,825},<<994>>,{986,150},{92,581},{988,147}}}

In[430]:= deleteDuplicatesBy[hugeList,1]//Short//Timing
Out[430]= {0.032,{{153,549},{887,328},{731,825},<<994>>,{986,150},{92,581},{988,147}}}

Comment: Perhaps `Union` and `SameTest` might also in some instances be useful?  For example `Union[{{1, 3}, {1, 2}, {4, 5}}, SameTest -> (First@#1 == First@#2 &)]`. I prefer Leonid's method, though.
Comment: wow, great! I did not know the criteria can be applied to 'whole' list, I thought it only applies to the specific 'element' at a time as it iterate. Learned something new. Thanks
Comment: @TomD `Union` with `SameTest` can also have severe performance issues for larger lists, please see my comments and links to a some relevant past discussions in the edit.
Comment: @Nasser The criteria is applied to two elements that are compared during a single comparison (application of the criteria), not to the entire list. In this case, the elements themselves happen to be lists (sublists of the main list), so we need to extract relevant parts to use in our criteria.
Answer: Leonid provided a long and thorough answer, as he often does.  However, I believe it is worth pointing out that an efficient and concise solution may be had with:
First /@ GatherBy[hugeList, #[[1]] &]
Where 1 is the column index to compare.
On my system this is faster than delDupBy but not as fast as deleteDuplicatesBy.


Question about pseudo-euclidean spaces and manifolds

Question: Is it possible to make a pseudo-euclidean space $ \mathbb{R}^{p, q} $ into a differentiable manifold?
My intuition tells me that yes since in special relativity we deal with the flat Minkowski space $ \mathbb{R}^{1, 3} $ however i tried to approach the problem of proving that for any given pseudo-euclidean space we can make it a differentiable manifold and i am stuck.
From what i understand, the way we prove this is by defining two charts $ (U, x) $ and $ (V, y) $  on local regions U and V in $M$ with two maps $ x: U \rightarrow A $ and $ y: V \rightarrow B $ where A and B are subsets of $ \mathbb{R}^{\dim(M)} $ and then looking at the transition maps $ y(x^{-1}(p)) $ to see if they are of class $ C^k $ and if they are for any pair of charts in M then M is a differentiable manifold
I tried to do this by defining a subset of $ \mathbb{R}^{1,2} $ as a collection of intervals.
$ U = ( (0;1); (1;2); (3; 4)) $
Which would mean my local region U would be from 0 to 1 in x, from 1 to 2 in y and from 3 to 4 in z.
And then i defined a map to be
$ \phi(p) = a_i p_i $
With $ a \in \mathbb{R} $ and then i did the same for V but with different intervals and different constants.
However when i evaluate the derivatives of the transition maps this is not $ C^k $
Help, i've literally never done this kind of exercises and now i'm stuck.
I don't know how to face this problem
Comment: Hint: One chart will suffice.
Answer: OK, let's try to make sense of your question. First of all, what is ${\mathbb R}^{p,q}$? Let $n=p+q$. Then ${\mathbb R}^{p,q}$ is the vector space ${\mathbb R}^{n}$ equipped with the quadratic form
$$
Q({\mathbf x})= x_1^2+...+x_p^2 - x_{p+1}^2- ... - x_n^2.
$$
In particular, as a topological space, ${\mathbb R}^{p,q}$ is just ${\mathbb R}^{n}$. (At this point it is critical that you know what the words "topological space" mean. I assume that you do.)
What does it mean to make ${\mathbb R}^{p,q}$ into a differentiable manifold? You want to accomplish two things:

Find a (maximal) smooth atlas on the topological space ${\mathbb R}^{n}$ such that the transition maps are smooth. (The "definition" that you wrote in your question is not quite the right one.) This will give ${\mathbb R}^{n}$ a structure of a differentiable manifold, I will call this manifold $M$.

Make sure that the quadratic form $Q$ becomes a semi-Riemannian metric on $M$ (of the signature $(p,q)$). (I assume that  you know what these words mean.)

Now, let's do it. As an atlas ${\mathcal A}$ on ${\mathbb R}^{n}$ I will take a single chart $(U,\phi)$, where $U={\mathbb R}^n$ (the entire space) and $\phi$ the identity map, $\phi({\mathbf x})={\mathbf x}$. Let's check the transition maps: Since there is only one chart, all what we have to check is that $\phi\circ \phi^{-1}$ and  $\phi^{-1}\circ \phi$ are smooth. But this is clear since both are just the identity map, so there is nothing to check here. Lastly, I will take a maximal smooth atlas containing ${\mathcal A}$ (this step is just a bookkeeping).  The result is a differentiable manifold structure on ${\mathbb R}^n$. This concludes Part 1.
For Part 2 you have to identify the tangent spaces $T_xM$ to our manifold $M$. Since $M$ is just ${\mathbb R}^n$, tangent vectors, elements of $T_xM$, are simply directed line segments $\overrightarrow{xy}$ with the tail at $x$ and the head at various points $y\in M$. I assume that in your vector calculus class you have learned how to identify the space of such segments with ${\mathbb R}^n$ itself (by computing the difference of coordinates of the points $x$ and $y$). With this identification, we define a quadratic form $Q_x$ on each $T_xM$ simply as our original form $Q$. If you want to be more explicit, for  $x=(x_1,...,x_n)$, $y=(y_1,...,y_n)$, we have
$$
Q_x(\overrightarrow{xy})= Q(y_1-x_1,,....,y_n-x_n)=
(y_1-x_1)^2 + ... + (y_p-x_p)^2 - (y_{p+1}-x_{p+1})^2 - ... - (y_n-x_n)^2.
$$
The function $(x,y)\mapsto Q_x(\overrightarrow{xy})$ is clearly smooth, hence, making our collection of quadratic forms $Q_x$ a semi-Riemannian metric on our manifold $M$.
Comment: Oh... you really need to retake calculus.... Or ask a new question on mayhstack exchange asking why constant functions are continuous.
Comment: Thank you for answering. Is a semi-Riemannian metric the same as a pseudo-Riemannian one? And how is the identity map smooth? Shouldn't its derivatives be 0 since it's constant?
Comment: Semi-Riemannian is the more standard name as pseduo-Riemannian. As for rhe derivative of the function $f(x)=x$, one usually learns this in a vector calculus class. Did you take one?
Comment: In order to succeed in (mathematical) physics, you absolutely have to have a solid math foundation, including at least calculus and linear algebra. If you did not take these classes or forgot what you have learned there, my suggestion is to take a class or read a textbook on your own.
Comment: I know that the derivative of $ f(x) = x $ is 1  but for the transition maps to be smooth we need for their derivatives to be continous. As far as i know $ F(x) = 1 $ is not a continuous function since for any x and small dx $ x \rightarrow x + dx $ does not induce a change $ y \rightarrow y + dy $ since y is a constant function


Do I have to use the name args in public static void main(String args[])

Question: I am new to Java and I'm learning it. I have a question regarding the statement
public static void main(String args[])

I understand all of it what it means. My question is regarding the String args[] part.
String args[] declares a parameter for method main() named args which is an array of instances of the class String.
So can I write anything in place of args[]? Like String pqrs[] or String abc[] or do I have to use args[] as a keyword?
I'm just curious to know how this works?
Comment: And conventionally, it would be `String[] args` as well. Putting the `[]` at the end of the variable name is an abomination :)
Comment: As always; the best wasy is to try it. But yes `args` is just a convention. You can call it whatever you want
Comment: Yes. It is just a variable name. You can use any name in place of "args"
Comment: @JonSkeet I really don't know why java even allows that. Its like putting half of the class name at the end of the variable *for no reason*
Comment: @RichardTingle: Familiarity for C/C++ developers, back when Java was very new. Unfortunate short-sightedness, IMO.
Answer: Using args is just a convention, all of the following would be fine and compile:
public static void main(String args[])

public static void main(String abc[])

public static void main(String anythingElse[])

However; there is annother (arguably more important) convention you are ignoring that will trip you up down the road.
Its usually
public static void main(String[] args)

Doing it the other way round will work, but is confusing because the general assumption is
TypeDefinition variableName

What is the type, is it a String? No, no it is not, its an array of strings, which we represent by String[]. Is the variable name args[]. No its just args
Comment: so by this convention is

(String[] abc, xyz) same as (String abc[], xyz[]) ?
Comment: @user3640307 Yes, and the first one is the correct way to write it
Answer: Yes, you can change the name to the one you want, because, at the end, the args name is just a convention.
The signature of the method public static void main, has an argument which is String[], and the name of the variable is up to you.
You can have either:
public static void main(String args[])

Or
public static void main(String arguments[])

Even this:
public static void main(String iAmAnArray[])

What you cannot have is something like this:
public static void main(List<String> iAmAList)

Because List is not part of the signature of the method main.
Answer: Short answer yes. I just tried it.
args[] is an array. So like most objects declared in a java application you have the freedom to name them.
Example
int i = 4;

I could change the "i" to "j" or "k" and it would still perform the same function.
I hope this clears that up for you.


Can the variance of two subsets of an observation of a random variable be greater than variance of the original complete set of observations?

Question: So if I have a finite set $ S \subset \mathbb{R} $ that can be partitioned in any form into two subsets $ S_1, S_2 $ such that $ S = S_1 \cup S_2 $ and $ S_1 \cap S_2 = \emptyset $.
Can I form $ S_1, S_2 $ such that $ Var(S_1) > Var(S) $ and $ Var(S_2) > Var(S) $ where $ Var(X) = E( (X - E(X))^2 ) $?
There are obvious cases if we only care about the variance of one set's variance, but I can't prove it (or find a case true case) for boths sets having greater variance. I every wrote a program to run through 1000000 random cases and did not find any true cases.
Revised question per discussion below accepted answer:
Per aiden's request, I believe a better formulation of his question is as follows:
If I have a set $S$ of observations of a real-valued random variable,  with variance $\sigma^2_S$, is there a partition of the observations into two disjoint subsets,  $S_1$ and $S_2$, where each subset has a larger variance than the original pooled set: $\sigma^2_{S_1} \geq \sigma^2_{S_2} > \sigma^2_S$?
Comment: I don't know what you mean by $\mathrm{Var}(S)$. Variance is defined for random variables, not subsets of $\Bbb R$...
Answer: It is assumed that the random variables are defined as equiprobable on the subsets of $\mathbb{R}$ on which they are defined.
$$\frac{1}{|S_1|}\sum_{z\in S_1}(z-\mu(S_1))^2+\frac{1}{|S_2|}\sum_{z\in S_2}(z-\mu(S_2))^2=Var(S_1)+Var(S_2)>\\>Var(S)=\frac{1}{|S|}\sum_{z\in S}(z-\mu(S))^2=\frac{1}{|S|}\sum_{z\in S_1}(z-\mu(S))^2+\frac{1}{|S|}\sum_{z\in S_2}(z-\mu(S))^2$$
We subtract.
$$\left(\frac{1}{|S_1|}-\frac{1}{|S|}\right)\sum_{z\in S_1}(z-\mu(S_1))^2-(z-\mu(S))^2+\\+\left(\frac{1}{|S_2|}-\frac{1}{|S|}\right)\sum_{z\in S_2}(z-\mu(S_2))^2-(z-\mu(S))^2>0$$
Now, each sum is nonpositive, and each leading coefficient is positive.  Hence together the left hand side is nonpositive, which is a contradiction.
Comment: Thank you. I am a bit concerned about claiming that each summation is non-positive. I believe I understand the intuition that the expected value of S will be farther from all z's that the expected value of S_1. But I wasn't sure if this was a valid argument.
Answer: I am understanding your question to ask if such a partition exists, and not does a set exhibit this behavior for all partitions. We know that:
$$
\begin{align}
Var(aX + bY) &= a^2 Var(X) + b^2 Var(Y) + 2abCov(X, Y)\\
&= a^2 Var(X) + b^2 Var(Y) + 2ab\rho_{X, Y}\sigma_X\sigma_Y
\end{align}
$$
Now $\rho$ can be -1. So if $X$ is negatively correlated with $Y$, we can create cases where  $Var(X \cup Y) \leq \min(Var(X), Var(Y))$
For example, try the following in R:
X <- seq(-4, 4)
Y <- -X
Z <- c(X, Y)
var(X); var(Y); var(Z)

You get:
[1] 7.5
[1] 7.5
[1] 7.058824

Update
If you do not allow multiple entries of the same value, try this:
X <- c(-1, 2)
Y <- -X
Z <- c(X, Y)
var(X); var(Y); var(Z)

[1] 4.5
[1] 4.5
[1] 3.333333

Comment: Two concerns. First, I think you meant to put Var(X union Y) in the 4th line ( Var(X \cup Y) in Latex). Also, this is clearly true, as you have a clear example. What then is wrong with vadim123's argument below?
Comment: What is meant by $X\cap Y$ (or $X\cup Y$ for that matter)? How do you apply a set operation to a function?
Comment: @Math1000, I am understanding the elements to be observations of one or more random variables. The question, as I understood it, was can there be a partition of the observation such that the variance of the elements of the subsets are both greater than the variance of the pooled observations.
Comment: @aidan.plenert.macdonald I'm not sure, but I do not see where vadim123 took the covariance of the two subsets into account in the second line of the first equation.
Comment: What do you mean by $\operatorname{Cov}(X,Y)$ when $X$ and $Y$ are sets? Also, please also give your examples in standard mathematical notation for the benefit of readers who don't know R.
Comment: @Rahul, under the assumption that the sets comprise elements which are observations of a random variable or variables, the covariance of the observations in the disjoint subsets for a particular partitioning. This does treat the elements as observations from two different variables. As for standard mathematical notation, please feel free to update the post if you feel it needs. I think the information is relatively clear, even if the notation is slightly abused, but others may differ.
Comment: I don't see how that makes sense. To define covariance of two random variables you need to know their joint distribution, and I don't know where you're getting that from. For example, say $X=\{0,1,2\}$ and $Y=\{-10,10\}$. What is $\operatorname{Cov}(X,Y)$?
Comment: And sorry, I should have been more clear. *I* don't know R so I don't understand your code. What is `seq`? What is `c`?
Comment: @Rahul, it doesn't make sense in as much as the original question is problematic; what is the variance of a set? So I chose to interpret it in light of what I believe was the intent of the OP; does there exist a set of observations of a random variable which can be partitioned so that the variance of each subset is > than that of the pool? The answer is clearly yes, that is how insurance works. I gave a specific example with same-sized sets that can be explained using covariance. I'd be surprised if there does not exist at least one dissimilar sized partition with similar properties.
Comment: @Rahul, `c` = concatenate. `seq` = sequence.
Comment: I apologize for any confusion in what I am asking, I come from a Computer Science background, as such I often don't know proper naming and notation conventions. Avraham has the right idea behind my question. I am asking about sets of observations of random variables. And I understand vadim123's answer below. He answered a slightly different question that what I was asking. Also, @avraham if you are willing, please correct my question to be mathematically proper. I simply wouldn't know how to do so.
Comment: @aidan.plenert.macdonald I added a revision to the question above; do you concur?
Comment: Looks good. Thank you


Binary floating-point representation of 0.1

Question: If $b=10$ and $p=3$, then 0.1 is represented as $1.00 × 1/10$. If $b=2$ and $p=24$, then 0.1 cannot be represented exactly, but is approximately
$$1.10011001100110011001101 × 1/2^4$$
Why do we divide by $2^4=16$ here?
Answer: The binary representation of $1/10$ is actually $0.000110011001100\ldots$, but since this is "floating point" we write this as $1.10011001100\ldots \times 2^{-4}$, just as in decimal floating point we would write $0.000123$ as $1.23 \times 10^{-4}$.  Each shift of the binary "decimal" point by one place corresponds to a factor of $2$.
Answer: How much is $1.10011001100110011001101_{\text{two}}$?
It's more than $1$. In fact it's clearly greater than $\frac32$.
And in any case it's quite a bit bigger than $0.1_{\text{ten}}$,
which is just $\frac{1}{10}$.
So you have to divide by something to get $\frac{1}{10}$ from $1.10011001100110011001101_{\text{two}}$.
What power of two would be sufficient?


Perform conditional calculations in a R data frame

Question: I have data in a dataframe in R like this:
Value   | Metric
10      | KG
5       | lbs       etc.
I want to create a new column (weight) where I can calculate a converted weight based on the Metric  - something like if Metric = "Kg" then Value * 1, if Metric = "lbs" then Value * 2.20462
I also have another use case I want to do a similar conditional calculation but based on continuous values i.e. if x >= 2 then "Classification" elseif x >= 1 then "Classification 2" else "Other
Any ideas that might work for both in R?
Answer: Does this work:
library(dplyr)
df %>% mutate(converted_wt = case_when(Metric == 'lbs' ~ Value * 2.20462, TRUE ~ Value))
  Value Metric converted_wt
1    10     KG      10.0000
2     5    lbs      11.0231

If you have other units apart from "KG" and "lbs" you need to include those in case_when condition accordingly.
Comment: I was hoping it would - it looked right but I got this error: Error: Problem with `mutate()` input `C_Weight`.
x non-numeric argument to binary operator
i Input `C_Weight` is `case_when(Metric == "lbs" ~ Value * 2.20462, TRUE ~ Value)`.
Comment: @MorneLippiatt,  Can you check the datatype of `Value` column? Is it numeric?
Comment: It is not... how can I change it?  I tried as.numeric but get another error saying it cannot be coerced to double.
Comment: I think I got it!  Used mutate(Value = as.numeric(DF$Value)) and that seemed to have worked.  Thanks!


Argument passed to call that takes no arguments error for Structs within Structs

Question: Hi all so I need to call a function that takes in the following parameters:
 let params = [
                "inputParameters" : ["A2":"a1","B1":"b1","C1":"c1","D1":"d1"]
            ]

so I made the following encodable struct for it
 struct Parameters2 : Encodable{

             let inputparameters = inputParameters(A1: "", B1: "", C1: "", D1: "")
            }

 struct inputParameters: Encodable {

                let A1: String
                let B1: String
                let C1: String
                let D1: String
            }

However when I call
guard let params = Parameters2(inputParameters(A1:"something", B1:"something", C1:"something", D1:"something"))
            else{
                return
            }

I get an error 'Argument passed to call that takes no arguments' even though I clearly defined the input arguments in the struct. Can anyone figure out what could be going wrong?
EDIT: Upon @vadian's suggestions I'm adding the function where this parameter is plugged into:
            func executePost(parameters: Parameters2, completion: @escaping (Any?, Error?) -> Void) {
            AF.request("myURL",
                       method: .post,
                       parameters: parameters,
                       encoder: JSONParameterEncoder.default, headers: headers).response { response in
                debug print(response)

                if let error = response.error {
                    completion(nil, error)
                }
                else if let jsonArray = response.value as? Any{
                    completion(jsonArray, nil)
                }
                else if let jsonDict = response.value as? Any{
                    completion([jsonDict], nil )
                }

            }}

And I call the function as follows:
  executePost(parameters: JSONEncoder().encode(params)) { (json, error) in <-- error Cannot convert value of type 'Data' to expected argument type '[String : InputParameters]'
                if let error = error{
                    print(error.localizedDescription)

                }
                else if let json = json {

                    print(json)

                }
            }

Comment: `let inputparameters = inputParameters(A1: ""...` -> `let inputparameters = inputParameters`? But class/struct names should start with an uppercase.
Comment: @Larme I tried that. But when I do that it says Parameters2 doesn't conform to 'Encodable' anymore :/
Answer: Your code doesn't work for several reasons:

A key mismatch inputParameters (dictionary key) vs. inputparameters (struct member).
The mentioned error which occurs because you assigned a default value in Parameters2.
guard let expects an optional, but there is none.

An easier way is one struct
struct InputParameters: Encodable {
      let A1, B1, C1, D1: String
}

and a literal dictionary key
let params = ["inputParameters": InputParameters(A1:"something", B1:"something", C1:"something", D1:"something")]

and encode params
Comment: Replace `parameters: Parameters2` with  `parameters: [String: InputParameters]`
Comment: I see. Interesting. So when you ask to encode 'params' do you mean to define another struct foe 'params'?
Comment: No. Just `let data = try JSONEncoder().encode(params)`
Comment: Gotchu. However my functoin requires the input to conform to encodable as follows: func execute(parameters: Parameters2) and I previously had Parameters2 conform to encodable. How would I manage that if I use JSONEncoder(). Am I able to explain what I'm trying to say? :/ Sorry I'm still learning these topics.
Comment: Because even when I use JSONEncoder I get 'Cannot convert value of type 'Data' to expected argument type 'Parameters2''
Comment: Please edit the question and add the code how you encode the object.
Comment: Just updated the question with the function that takes in 'params'
Comment: Okay, that eliminates the error form the function but when calling it by using JSONEncode I still have the error 'Cannot convert value of type 'Data' to expected argument type '[String : InputParameters]''  near '.encode(params)'
Comment: cuz when I use 'try' next to JSONEnocder, my button that executes the function throws error 'Invalid conversion from throwing function of type '() throws -> Void' to non-throwing function type '() -> Void''
Comment: No, AF encodes the parameters implicitly. Call the method `executePost(parameters: params)`
Comment: Yes, that works. Thanks, man! Appreciate the help. Now I'm back to my trouble of parsing Json and hopefully, I can figure that out lol.
Comment: Decode the stuff with `responseDecodable`.
Comment: Holy cow how I did miss that in the docs. Real quick: would my Response type be 'String' or 'encodable'? I'm kinda confused on the docs response type references. Regardless, thanks for the suggestion!
Comment: Hmmm interesting. Looks like I need to create a struct for the kind of data I'm expecting in the response and them set that as the Decodable type.
Comment: Hey @vadian, I was able to get the struct and everythingworking however everytime I decode it, it returns nil. For some reason my jsonData is being shown as ' XXbytes' when I parse it as Data. Do you think you'd be able to help me privately with this?
Comment: Please ask a new question and show the relevant information.
Comment: I asked it here: https://stackoverflow.com/questions/75142639/jsondecoder-returning-nil-while-parsing
Comment: Why do you ask two almost identical questions? I suggested a solution in your other question.


Difference between dummy and factor variable?

Question: I've just learnt about dummy variables. Say this is my data:

Location
Nest

XXX
Yes

XXX
No

ZZZ
Yes

YYY
Yes

YYY
No

And I want to do multicolinearity tests/logistic regression in RStudio, so I don't want the dependent variable (Nest) to be in this text format.
What is the difference between changing all "no" to 0, and all "yes" to 1 versus having the below output (which, as I understand, is the 'dummy' encoded version of the above data).

Location
Nest
No nest

XXX
1
0

XXX
0
1

YYY
1
0

Moreover, if I have say 15 categories that I want to analyse, can I just label them #1-15, or do I need to have 14 columns (for the k-1 I suppose) to make the categories into dummy variables?
Comment: You don't need to recode the Location or Nest variables or worry about dummy variables in R or Rstudio. R will create dummy variables for you automatically when you include the variables Location or Nest in a linear model or glm.
Answer: You are confusing few things.

There is no such data coding type as "factor". The factor is an R's internal data representation type. It is R's legacy from many years ago when computers had much less memory available. To overcome the memory issues, R's predecessor, the S language, introduced the factor data type for categorical variables. It encodes the values as numbers but displays them using human-readable labels. A binary variable having two values "Yes" and "No" when encoded as factor would in fact be coded using the numerical values of 0 and 1. When you use it, under-the-hood R decodes it into dummy encoded variables when needed.
Your second table shows a variable encoded using the one-hot encoding. In such encoding, every level of the variable gets its own column, so for $k$ categories, you end up with $k$ columns.
Another way of coding is to use dummy encoding, where we drop one of the columns so there are $k-1$ columns. We can do this because one of the columns is redundant.

For learning more about the differences between the two latter categories check the One-hot vs dummy encoding in Scikit-learn thread.


Java's SecretKeySpec in Ruby?

Question: I'm trying to port over some Java code to Ruby. I was given a class that uses several classes in Java's crypto package (javax.crypto.*) in order to encrypt a string using Blowfish. There are several parts that I'm stuck on and would love some help.
public String encrypt(String var1) throws BlowfishCipher.BCException {
    try {
        byte[] var2 = var1.getBytes("UTF-8");
        byte[] var3 = this.encrypt(var2);
        return bytesToHexString(var3);
    } catch (Exception var4) {
        if(var4 instanceof RuntimeException) {
            throw (RuntimeException)var4;
        } else if(var4 instanceof BlowfishCipher.BCException) {
            throw (BlowfishCipher.BCException)var4;
        } else {
            throw new BlowfishCipher.BCException(var4);
        }
    }
}

public byte[] encrypt(byte[] var1) throws BlowfishCipher.BCException {
    try {
        Cipher var2 = Cipher.getInstance("Blowfish");
        byte[] var3 = hexStringToBytes(this.sKey);
        SecretKeySpec var4 = new SecretKeySpec(var3, "Blowfish");
        var2.init(1, var4);
        return var2.doFinal(var1);
    } catch (Exception var5) {
        if(var5 instanceof RuntimeException) {
            throw (RuntimeException)var5;
        } else {
            throw new BlowfishCipher.BCException(var5);
        }
    }
}

private static String bytesToHexString(byte[] var0) {
    StringBuffer var1 = new StringBuffer();

    for(int var2 = 0; var2 < var0.length; ++var2) {
        String var3 = Integer.toHexString(255 & var0[var2]);
        if(var3.length() < 2) {
            var1.append("0");
        }

        var1.append(Integer.toHexString(255 & var0[var2]));
    }

    return var1.toString();
}

Again, I can do most of it in Ruby, but the thing that I'm missing is the SecretKeySpec. Is there an equivalent in Ruby?
My code in Ruby looks something like this. The difference is that the Cipher I'm newing up is blowfish instead of bf-ecb
OpenSSL::Cipher.new('blowfish').encrypt

I also set my own key_length because the key that I'm supposed to be using is 14 characters.
Comment: Note that ECB is not secure, and that Blowfish is a cipher with an 8 byte block size, which is also not that secure. If you use encryption for realtime transfer you also need integrity protection / authentication (e.g. using a HMAC). Update you protocol ASAP!
Comment: Any progress after the initial success for the first block, snguyen? It may be another mode of operation, but ECB and CBC are generally the prime suspects.
Comment: $ ruby cipher.rb
cipher.rb:5:in `initialize': unsupported cipher algorithm (blowfish-112-ecb) (RuntimeError)

$ ruby cipher.rb
cipher.rb:5:in `initialize': unsupported cipher algorithm (blowfish-112-cbc) (RuntimeError)
Comment: Yeah, OK, so maybe you have to construct it differently, providing the blowfish and cbc parameters separately. I saw quite a few ways of constructing ciphers(which has a variable key size).
Comment: I meant blowfish has a variable key size, of course.
Answer: The Java code seems to hex decode the key first. The SecretKeySpec is just a wrapper around the (random or pseudo-random) bytes of the key.
So you first need to decode the hexadecimal string and then use that as key:
cipher = OpenSSL::Cipher.new('blowfish-xxx-ecb')
cipher.encrypt
cipher.key = [sKey].pack('H*')
return cipher.update(var1) + decipher.final

Where xxx is the size in bits of the decoded key.
Comment: Note that I'm not a Ruby developer, so please correct syntax errors in above using the edit button.
Comment: So this gets me very close! When I print out the byte array in Java, I get [31, -4, -64, 115, 5, 35, -124]. When I print out the byte array in Ruby, I get [31, 252, 192, 115, 5, 35, 132]. The only difference is, in Java, there are negatives, and in Ruby, there are not.
Comment: Java bytes are always signed (which is actually quite irritating). So bitwise both strings would be identical. I.e. you should be good. Print each byte out after performing `& 0xFF` to see the positive values.
Comment: Ah, crap. The encrypted results are VERY close. Ruby: cf2c2baca6f890630fee34dcdcc0fdaa. Java: cf2c2baca6f890634a3097d20933a84b
Comment: Did you use a specific library / provider for the Blowfish implementation in Java? If yes, which one? Please try `blowfish-xxx-cbc` instead. If that doesn't work, try and set the `cipher.iv` to 8 zero bytes explicitly (16 hex `0`s, you know how to decode now).
Comment: So I'm not sure what the 'xxx' should be. If my key is 14 characters long, is it 'blowfish-14-cbc'? If so, I get `initialize': unsupported cipher algorithm (blowfish-14-cbc) (RuntimeError)
Comment: 14 * 8 = 112 I guess, it's bits, not bytes. Don't think of bytes as characters, even if the language doesn't distinguish between the two. Or 56 if you talk about the key size in *hexadecimal digits*.
Comment: When attempting to encrypt 'mystring', the encrypted string starts to get different at the half way point. I decided to try to encrypt the string 'my', and got the same results between the Java and Ruby version. 'mys' also encrypts to the same string in Ruby and Java. They start diverting when I add 't' ('myst') to the string to encrypt.
Comment: That's very strange. I'll try and think about this, but I'm not sure I understand. Maybe different modes like CFB-24 or something like that, but that would be a very long shot.

Java's SecretKeySpec in Ruby?


how to extract the utm_content from the URL in SQL bigquery

Question: I'm trying to extract the utm_content from the URL.
For example:
https:/www.xxxxxx.com/br/pt/yyyyy/example?origin=ALL&destination=ALL&utm_source=salesforce&utm_medium=email&utm_campaign=just_an_example_ok&utm_content=I_would_like_just_this_part_of_the_string
¿How could I extract just the utm_content after the utm_content=?
Generally, there is  12-13 fragments (splited by _) in my utm_content.
I would like to have this:

URL
utm_content

https:/www.xxxxxx.com/br/pt/yyyyy/example?origin=ALL&destination=ALL&utm_source=salesforce&utm_medium=email&utm_campaign=just_an_example_ok&utm_content=I_would_like_just_this_part_of_the_string
I_would_like_just_this_part_of_the_string

Any suggestions?
Comment: mysql or bigquery?
Answer: See example below
<code>select url,
  regexp_extract(url, r'utm_source=([^&]+)') as utm_source,
  regexp_extract(url, r'utm_content=([^&]+)') as utm_content
from your_table
</code>
if applied to sample data in your question - output is
Comment: Thank you! It worked as I expected.


Strange behavior of Control.Page in nested ViewUserControl

Question: I have a ASP.Net MVC view which contains other partial views derived of ViewUserControl. The view itself is contained in a master page, so the control hierarchy can be summarized as follows:

    Master page              - basic settings
       ViewPage              - "the" page
          ViewUserControl    - content component 1
          ViewUserControl    - content component 2
          ...
          ViewUserControl    - content component n

Additionally, we have a custom control for the declaration of CSS and Javascript files. This custom control ensures that, no matter how deep into the hierarchy, whenever there is a CSS or JS dependency it will be rendered inside the Page.Header, thus avoiding scattered and duplicated references. Let's call this custom control <my:dependency>.
All is well when <my:dependency> is hosted in the master or ViewPage. But if I use it deeper inside the content components, Page.Header is null. Debugging revealed that, for controls hosted in a ViewUserControl, their Page is the very ViewUserControl in which they're in, and so the Header is indeed missing.
How can I make <my:dependency> "see" the topmost ViewPage and get access to the Header? Thanks.
Answer: Now I could be wrong, but you appear to be trying to get a web forms control to work in MVC, something that is not recommended for reasons you've discovered.
You might want to look at something involving the ViewData collection.
Simon
Comment: I realized this some time after posting the question, and decided to approach the problem differently.


$this->url returns only the module path

Question: I'm using Zend Framework 2.2.4 to create an mvc application.
I have a module called DrinkManangement and inside of it i have a controller called DrinkController and an action called drinkQueryAction.
from a view phtml called add-inventory i'm trying to add the url to a form action attribute.
i'm using the following code:
$drink_query_form->setAttribute('action', $this->url('drink', array('action' => 'drink-query')));

the output of the $this->url command only returns /drink-management that is the path of the module. but it doesn't return /drink-management/drink/drink-query.
what am I missing here ?
thanks
update
ok it's probably related to my route configuration but i have no clue how to fix it
'router' => array(
    'routes' => array(
        'drink' => array(
            'type'    => 'Literal',
            'options' => array(
                // Change this to something specific to your module
                'route'    => '/drink-management',
                'defaults' => array(
                    // Change this value to reflect the namespace in which
                    // the controllers for your module are found
                    '__NAMESPACE__' => 'DrinkManagement\Controller',
                    'controller'    => 'Drink',
                    'action'        => 'add-drink',
                ),
            ),
            'may_terminate' => true,
            'child_routes' => array(
                // This route is a sane default when developing a module;
                // as you solidify the routes for your module, however,
                // you may want to remove it and replace it with more
                // specific routes.
                'default' => array(
                    'type'    => 'Segment',
                    'options' => array(
                        'route'    => '/[:controller[/:action]]',
                        'constraints' => array(
                            'controller' => '[a-zA-Z][a-zA-Z0-9_-]*',
                            'action'     => '[a-zA-Z][a-zA-Z0-9_-]*',
                        ),
                        'defaults' => array(
                        ),
                    ),
                ),
            ),
        ),
    ),
),

Comment: What's the definition of the route `drink`?
Comment: updated main post with route configuration
Answer: You should understand first how routes work. You have a "root" route called drink. This route has a single child called default. Now is drink a literal route. That means, there are no parameters in this route and it's always a static url /drink-management.
The child-route default is a segment route, which you can give parameters. So if you use the child-route in a view helper, you can assign parameters with an array like array('controller' => 'foo').
Now, the important part is you can point to this child route by using / and its parent. So your default can be reached via drink/default.
A TL;DR:
$this->url('drink/default', array('action' => 'drink-query'));

PS. A comment on your naming: I would advise you to rename your routes so they make more sense. In a way, drink is your route and that's fine. The child route default is confusing as it is no default in general, but just a catch-all for your drink controller. It makes more sense to merge it into the drink route. Then you have a single route where you can specify an action (and the controller just remains the same):
'router' => array(
    'routes' => array(
        'drink' => array(
            'type'    => 'segment',
            'options' => array(
                'route'    => '/drink-management[/:action]',
                'defaults' => array(
                    '__NAMESPACE__' => 'DrinkManagement\Controller',
                    'controller'    => 'Drink',
                    'action'        => 'add',
                ),
            ),
        ),
    ),
),

Now you have a drink route which you can give an action and it just stays on the drink controller:
$this->url('drink'); // to /drink-management

Or an action:
$this->url('drink', array('action' => 'query')); // to /drink-management/query

$this->url returns only the module path


Alterar a escala do eixo X

Question: Olá, estou tentando fazer um gráfico de afilamento do tronco de uma árvore, onde o eixo y é a altura (metros) e o eixo x é o diâmetro (cm). Até aí sem problemas, porém gostaria de mudar a escala do eixo x para dar mais realismo e não estou conseguindo fazer isso, alguém teria alguma sugestão ?
Comment: O que você chama de mudar a escala do eixo x, transformar de cm para metros por exemplo?
Answer: Se tu estiver usando os comandos básicos do R para fazer o gráfico (plot()), tu pode mudar o eixo x/y com o parâmetro xlim/ylim
x <- rnorm(100, 10, 2)
y <- rnorm(100, 100, 20)
plot(x, y)
plot(x, y, xlim = c(0, 120)) # Alterando somente o eixo X
plot(x, y, ylim = c(0, 120)) # Alterando somente o eixo Y
plot(x, y, xlim = c(0, 120), ylim = c(0, 120)) # Alterando os 2 eixos

Comment: @MarcoMachado O que você quer então? Edite a sua pergunta para deixar o problema mais claro se esta não é a solução que você está procurando.
Comment: Obrigado pela sugestão, mas eu havia tentado fazer isso.. Até faz o que eu quero o problema é que o eixo x ou y, fica demasiadamente extenso e sem utilidade no espaço vazio..
Answer: Eu segui esse tutorial, que mostra como configurar os eixos de uma maneira bem didática.
Mas em resumo, explicando com minhas palavras e da forma que usei no meu caso (sou iniciante no R), eu fiz assim:
# axes=F -> retira os eixos do seu gráfico.
plot("seus dados", ..., axes=F)

# A função axis permite configurar os eixos, sendo "1"
# indicativo do eixo "abaixo" do gráfico (o eixo "x")
# e o parâmetro "at" indica que valores devem aparecer
# no eixo. Da mesma forma para o eixo vertical.
axis(1,at=c(x1,x2,x3,...,xn)

# "2" indica o eixo y (à esquerda do gráfico)
# e pelo "at" seus respectivos valores.
axis(2,at=c(y1,y2,y2,...,yn))

Comment: Obrigado!!!!!!!

Alterar a escala do eixo X