Why can't I get result list, from a query?

Question: I want to login my users only if their account is activated, but for some reason the query that should return me the user (I call it Role in my program) when given the correct credentials, does not retrieve anything at all.
I am 100% sure that the credentials are correct, also I'm sure that there are no duplicates in the database and the query syntax is correct.
So why doesn't this method inside one of my EJBs returns true?:
// Checks if the account is not activated!
    public boolean isActivated(String email, String password) {
        // 1-Send query to database to see if that user exist
        Query query = em
                .createQuery("SELECT r FROM Role r WHERE r.email=:emailparam AND r.password=:passwordparam");
        query.setParameter("emailparam", email);
        query.setParameter("passwordparam", password);

        List<Object> tmpList = query.getResultList();
        if (tmpList.isEmpty() == false) {
            System.out.println("IS NOT EMPTY");
            Role role = (Role) tmpList.get(0);
            if (role.getAccountStatus().trim()
                    .equalsIgnoreCase(AccountStattus.ACTIVATED.toString())) {
                // The account is activated!
                System.out.println("ACTIVATED!!!!!!!");
                return true;
            }
        }
        // The account is not activated!
        System.out.println("NOT ACTIVATED!!!!!!!");
        return false;
    }

The only message i see in the console is NOT ACTIVATED!!!!
Update
To comfirm that the list is empty at 100% I modified my code to look like this:
// Checks if the account is not activated!
    public boolean isActivated(String email, String password) {
        // 1-Send query to database to see if that user exist
        System.out.println("HERE:" + email+password);
        Query query = em
                .createQuery("SELECT r FROM Role r WHERE r.email=:emailparam AND r.password=:passwordparam");
        query.setParameter("emailparam", email);
        query.setParameter("passwordparam", password);

        List<Object> tmpList = query.getResultList();
        Iterator<Object> it = tmpList.iterator();
        int elements = 0;
        while(it.hasNext()) {
               elements++;
             }

        if (elements > 0) {
            System.out.println("IS NOT EMPTY");
            Role role = (Role) tmpList.get(0);
            if (role.getAccountStatus().trim()
                    .equalsIgnoreCase(AccountStattus.ACTIVATED.toString())) {
                // The account is activated!
                System.out.println("ACTIVATED!!!!!!!");
                return true;
            }
        }
        // The account is not activated!
        System.out.println("NOT ACTIVATED!!!!!!!");
        return false;
    }

The method still returns false when I enter the correct credentials. What is wrong?
Comment: What type is "r FROM Role"? May your query return nothing?
Comment: Did you check which SQL statement is created and did you test that one directly on the DB?
Comment: The name of the table is "Role". The return statement my return nothing, but i am giving good credentials and i don't understand why it does not return true
Comment: @The Elite Gentelman I did what you say but this is what the console says: `Syntax error parsing the query [SELECT * FROM Role r WHERE r.email=:emailparam AND r.password=:passwordparam], line 1, column 7: unexpected token [*].`
Comment: @sfrj, sorry, my mistake. I confused SQL with JPQL. I deleted my comment.
Comment: there is nothing like *. try with this `SELECT r FROM Role  r`
Comment: I am using `r`, i am not using `*` but does not work. I also just checked the credentials(user6 user6), there is just one Row in the DB with those credentials. I don't understand why it does not return true.
Comment: Can you check the generated SQL query ?
Comment: What do you mean by check the generated SQL?, this is how it looks like: `SELECT r FROM Role r WHERE r.email=:emailparam AND r.password=:passwordparam`
Comment: @sfrj: Jigar means the SQL query (not the Hibernate query). There must be a hibernate properties file containing a param: hibernate.show_sql and it should be `hibernate.show_sql=true`. That will print the executed query.
Comment: Anyway I would try removing the "SELECT r" from the query. Simply: `FROM Role r WHERE r.....`
Comment: Hey, just chiming in to remind @sfrj and the rest of the world to never store plain-text passwords. I hope you are doing at least some sort of salt+hash on the password and storing that instead of the password in plain text.
Answer: Steps to debug,

Run the same query using any SQL client
Check your Role class. It must have a email and password properties with appropriate getters/setters

And please don't tell me that the password in the database is a hashed-version.
Few basic refactoring suggestions,

Write your query in this way to make it easy on eyes,
Query query = em.createQuery("from Role r where r.email=:email and r.password=:password")
                .setParameter("email", email)
                .setParameter("password", password);

Change your condition to this,
if (!list.isEmpty()) {...}

Comment: @sfrj: Glad it worked. But why `password` parameter got lost, and not the `email`? Just curious.
Comment: Nothing special. I use a Managed bean to get the password from the front end and and handle it over. I did some string manipulation to remove blank spaces, but i did not trim() at the end. So the password was wrong. I could not see it was not trimmed in the console since i cannot see blank spaces at the end of the words :)  A bit silly, from me, probably while i was refactoring yesterday i created the problem.
Comment: I fixed my problem after a bit of debugging. It looks like the password parameter got lost on the way. The query was fine but the credentials wrong when arrived to the EJB. Thank you!
Answer: Your query list must be empty.  It's either a problem with the database you're connected to (perhaps not the one you think you're tapping), the query, the parameters, or the database.
Comment: Yes the list is empty, that for sure since the condition `if (tmpList.isEmpty() == false)` is not evaluated to true. I am sure at 100% that the database is correct, because i can correctly register users and i see the data inserted in the DB.


Intersection Dense and Regular Closed Sets is Dense

Question: Let $X$ be a metric space.  It seems intuitive to me that if $D$ is dense in $X$ and $K=cl(int(K))$ (ie: $K$ is a regular closed set in $X$) then
$$
cl(D\cap K) = K,
$$
that is $D$ is dense in $K$....but how to show this/is it actually true?
Answer: $int(K) \subset cl(D\cap K)$ because if $x \in int(K)$ and $U$ is any open set containing $X$ then  $U\cap int(K)$ is a nonempty open set; so it intersects $D$. Thus every neighbourhood of $x$ intersects $D \cap K$ proving that  $int(K) \subset cl(D\cap K)$. Now just take closure on both sides to see that $K \subset cl(D\cap K)$. The reverse inclusion is obvious.
Comment: Thank you Kavi.  Though I'm curious, does the argument still hold if we consider relative interiors and relative closures?
Comment: I don't quite understand this question. Relative to what?
Comment: Say, relative to a closed subspace $\tilde{X}$ of $X$.
Comment: Yes, the result is still true and you only have to replace the metric space $X$ by the closure of $K$ in the proof.
Comment: Awesome!  Thanks so much Kavi :)


netperf + iptables masquerade -> network unreachable

Question: Why iptables rules pass netperf TCP_STREAM test through, but break UDP_STREAM?
I have a network:
    +---------------+
    |               |
[client]--[NAT]--[server]

On the NAT, I have added the following iptables rules:
$ iptables -t nat -L:
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

$ iptables -L:
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

It works well, when I run ping server on the client, when I run netperf through TCP, and when I run netperf through UDP via the direct connection. But it does not work if I run:
$ netperf -H 192.168.2.10 -t UDP_STREAM -l 1
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.10 () port 0 AF_INET : demo
send_data: data send error: errno 101
netperf: send_omni: send_data failed: Network is unreachable

It seems, the iptables firewall blocks the UDP packets.
Is it so, and how do I configure it not to?
Answer: The answer is here: https://stackoverflow.com/a/24211455/1234026
In short: netperf disables IP routing by default for UDP_STREAM test, and if the target address is in a different subnet, it fails to find the route. To make it act normally, I need to supply -R 1 key as a test-specific option.


How to record unique hits in node counter stats?

Question: Is there a way to only count the unique views in the core node counter stats?
I was using the Statistics Advanced module for Drupal 6 but there is no Drupal 7 solution. The quote below is from the Statistics Advanced module and what I want exactly is that.

Only track 'unique' views of content: If you keep viewing a node over
  and over again, the "x reads" counter won't increase each time. This
  is applied on a per-user basis as well as a per session id for
  anonymous users.

The Node View Count module doesn't effect the core statistics, it creates a new table for itself. So this is not an option for me.
What I'm looking for is a solution that effects the CORE statistics, nothing else.
Comment: Are you sure the other module does what you want? It says: "Unlike statistics module, this module records each view of node with user uid and timestamp. So now you can know how many times a user viewed node by day, week, month and etc." I take it counts every visit of the same user for the same node, not that it counts a single visit per day.
Comment: Anyway, the _Node View Count_ project page states, for 7.x-2.x: "Also now we count node views for anonymous users."
Comment: @kiamlaluno, the other module (Node View Count) doesn't do what I want as I said. But I missed that you wrote on your 2nd comment and I will check it. Thanks.
Comment: Node View Count should do the trick, however if you are going to use this to list popular articles, keep in mind if you do it with views default query is very inefficient and can slow down your site as it executes too oftern, better off writting own custom module to do sorting.
Comment: @pgrujic, is there any comparison or known performance report for the Node View Count that show it's inefficient to use. Thanks.
Comment: UNIQUE in terms of what ? in terms of IP ? if not users ?
Comment: @BipinK, in terms of IP address or session, or something else that can be used for uniqueness.
Answer: Node view count
You may want to use the Node view count module for this. Here is a quote about it (from its project page, bold markup added here):

... allows to count node views. Unlike statistics module, this module records each view of node with user uid and timestamp. So now you can know how many times a user viewed node by day, week, month and etc.
This module is integrated with Views module. You can use it to create different statistics pages, output new not viewed nodes for user and many more.

Specific to the D7 version of it, this is what is included also on the project page:

Also now we count node views for anonymous users.

Statistics Advanced
Be aware of the  Statistics Advanced (*) issue "Port Statistics Advanced Settings to D7", with current status = needs review.
My "recommendation" for anybody looking for a D7 version of it, is to review what is attached to Comment # 55 in that issue. And report any potential deficiencies in that same issue (confirmations that it works fine would be appreciated too, also via that same issue).
Other modules to consider
If the Node view count and/or Statistics Advanced (*) module doesn't fit your needs, then you may want to consider reviewing this shortlist of possible alternatives (quotes are from their project pages).

Visitors (*):

Displays information about visitors, and integrates with the WhoIs Lookup module

Better Statistics:

introduces a Statistics API, which allows for a way to collect, limit, and route custom access statistics and data.

Piwik Web Analytics:

Adds the Piwik web statistics tracking system (free open source) to your website.
Related module: Piwik Reports, which provides graphical reporting of your site's tracking data inside Drupal.

Chartbeat:

... to provide third party integration with Chartbeat, for real-time site monitoring statistics.

Stat Counter:

... provides an integration between Drupal and StatCounter web statistics.

Enough options to "explore"?
Disclosure: I'm the maintainer of the modules marked with (*).
Comment: Thanks for your answer. I will use the Node view count, Better statistics and Visitors module and share the results. Statistics Advanced module is only for Drupal 6, unfortunately (as I said I was using it before). And the other modules are 3rd party integration modules and actually I don't want to use these kind of modules. Thanks again.
Comment: "**M**erci" to "**H**erci"! And note the extra addition to my answer I just did for the Statistics Advanced module. Are you interested in helping with the "review" of that "issue with comment nr 55" I suggested?
Comment: OK, after a long time I tried all these modules... but unfortunately none of them does what I really want. What I need is a simple solution that effects the **core** statistics (module) and tracks only unique hits (based on uid, IP address or session).
Answer: You can try the Better Statistics module.
On the module page it says:

This module introduces a Statistics API, allowing developers and
  contributed modules a way to collect, limit, and route custom access
  statistics and data.
For Administrators and Site Builders

Customize what data is collected on each page visit
Limit where and for whom data is collected
Use Views (of type "access log") to generate custom reports using    your custom data
Download modules that add and extend the above functionality


Why dplyr case_when doesnt work correctly?

Question: I am working on a data frame and I want to generate a new variable conditionally based on values, but those values greater than 1 are supposed to say "Sin sentido", however, this is not the case and it still appears "En desarrollo", Does anyone know why this happens? Is there an error in the syntax of my code? Hope you can help me
The code is the following:
ejercicio <- data.frame(
  dato1 = c(0,0.1,0.5,1,1.1)
)

ejercicio <- ejercicio |>
  mutate(Estatus = case_when(
    dato1 == 1 ~ "Completado",
    dato1 == 0 ~ "Sin empezar",
    dato1 > 0 | dato1 < 1 ~ "En desarrollo",
    dato1 > 1 ~ "Sin sentido"
  ))

Comment: Jorge, as highlighted in the answer, the "OR" is valid for all values. If you want to check for the thresholds, aka == 0 / 1, and what is between, your logical condition should be the "AND" (i.e. & := dato1 > 0 & data1 < 1) rather than an OR.
Answer: The reason is that the third condition already meets for all the rows
 with(ejercicio, dato1 > 0 | dato1 < 1)
[1] TRUE TRUE TRUE TRUE TRUE

Thus, the last condition is not evaluated.  Probably, we could change the sequence of expression i.e. the last should go above the third condition
ejercicio |>
  mutate(Estatus = case_when(
    dato1 == 1 ~ "Completado",
    dato1 == 0 ~ "Sin empezar",
    dato1 > 1 ~ "Sin sentido",
    dato1 > 0 | dato1 < 1 ~ "En desarrollo"

  ))

-output
dato1       Estatus
1   0.0   Sin empezar
2   0.1 En desarrollo
3   0.5 En desarrollo
4   1.0    Completado
5   1.1   Sin sentido

NOTE: These are floats, so using == should be careful i.e. it may not be exactly equal unless precision is taken care of


Cannot create .Net Core dev certificate on macOs

Question: I'm able to successfully build my solution in Visual Studio however when I attempt to run, I'm seeing HTTPS development certificate not found
When I click Install and Trust I get a "dotnet dev-certs wants to make changes" popup. Soon after  I enter my password my project quits. When I try to run dotnet dev-certs https --trust in my terminal I get a "A valid HTTPS certificate is already present" even though I've deleted all localhost certs in my keychain. I've tried running a dotnet dev-certs https --clean, and I've tried unlocking, locking, and unlocking my keychain login again but no luck.
I'm on dotnet 5.0.100 and currently on macOs Catalina 10.15.7
Answer: You can always work without a certificate :

Go to your project's options
Go to the "ASP.NET CORE" tab
under Run > Configuration, Remove "https://localhost:5001", leave
http://localhost:5000

You should be able to work with the http://localhost:5000 URL and thus not worry about the certificate.
Comment: Went and gave that a try and now I'm just receiving a 404 or the project quits. Changed the url and appended '/api' at the end but no luck.


db_insert in Drupal 7 without field names

Question: I have the following INSERT in a db_query, but feel like I should convert it to a db_insert, but can't figure out the syntax:
INSERT INTO {mytable} VALUES (23, 'foo', 'bar', 42)

This example is fairly short, but I do work with some legacy tables that have MANY fields.
$insert = db_insert('variables')->values(array(23, 'foo', 'bar', 42));
$insert->execute();

The above gives me "NoFieldsException: There are no fields available to insert with. in InsertQuery->preExecute()".
How can you perform an insert with db_insert without needing to specify each field name?
I can make it work in D7 with db_query([INSERT...]); but drupal_write_record does not work.
Answer: You can not do this with db_insert().  You could probably dig into the InsertQuery class and get it done, but that's no different than what you're already doing IMO.
I have never seen INSERT without specifying column names, for good reason, although apparently you can (mysql api).  If you do this, it makes it more difficult for someone else to understand what you've done, and impossible to alter the table structure without causing headaches.
You can look at the waydrupal_write_record() uses drupal_get_schema() to create the fields if you'd like to automate the process (although it would probably take just as long to write the $fields array one time).
Comment: Thanks.  I was afraid that was the case, and probably is much more time to get it to work.  INSERT without columns is standard SQL, but like you said if someone changes the table schema it would be a headache.  The table I am using is legacy and chiseled in stone at this point :)


Intercept tls.Conn's Read() and Write() while retaining tls.Conn functionality

Question: I have the following piece of code:
l, err := tls.Listen("tcp", "localhost:0", cfg)
dieIf(err)
c, err := l.Accept()
dieIf(err)
err = c.(*tls.Conn).Handshake()
dieIf(err)

It works just fine, but I'd like to intercept tls.Conn's reads and writes.
I thought about doing this:
type MitmConn struct {
  net.Conn
}

func (self *MitmConn) Read(b []byte) (n int, err error) {
  ...
}

func (self *MitmConn) Write(b []byte) (n int, err error) {
  ...
}

l, err := tls.Listen("tcp", "localhost:0", cfg)
dieIf(err)
c, err := l.Accept()
dieIf(err)
c = &MitmConn{c}

But then, this would fail:
// panic: interface conversion: net.Conn is *MitmConn, not *tls.Conn
err = c.(*tls.Conn).Handshake()
dieIf(err)

Any ideas?
Comment: `c.(*MitmConn).Conn.(*tls.Conn).Handshake()`
Answer: package main

import "crypto/tls"

func dieIf(err error) {
    if err != nil {
        panic(err)
    }
}

type mitmConn struct {
    *tls.Conn
}

func (mc *mitmConn) Read(b []byte) (n int, err error) {
  return 0, nil
}

func (mc *mitmConn) Write(b []byte) (n int, err error) {
  return 0, nil
}

func main() {
    l, err := tls.Listen("tcp", "localhost:0", nil)
    dieIf(err)
    c, err := l.Accept()
    dieIf(err)
    mc := mitmConn{c.(*tls.Conn)}
    err = mc.Handshake()
    dieIf(err)
}

Please consider unlearning naming the receivers self (and also this).
Comment: Thanks for the tip and the response. This doesn't intercept the handshake's writes and reads, though. It just intercepts any explicit reading or writing (AKA mc.Write() and mc.Read()), but not the activity happening inside (tls.Conn.Handshake())
Comment: @solidak, I'm afraid, unfortunately, you won't be able to intercept reads and writes during the handshake: specifically because there's no "inheritance in the C++ sense": even if you embed a value of the correct type in your "MitM type" so that it compiles, the `Handshake` method gets promoted from the embedded type to the enclosing type, but when that method gets called on a value of the enclosing type, the receiver passed to that method will be of type `*tls.Conn`—that is, the value of the embedded type, not the enclosing type.
Comment: @solidak, specifically see [this section of the spec](https://golang.org/ref/spec#Struct_types): «If S contains an embedded field *T, the method sets of S and *S both include promoted _methods with receiver `T` or `*T`»_ (emphasis mine). Basically this means that when you call `Handshake` on an instance of your "MitM type", it really gets called on the embedded field of type `*tls.Conn`, and hence `Read` and `Write` will be called on that same receiver.
Comment: I hear. You're correct. I had to do a "hacky" way by basically breaking open tls.Accept(). The example I had is not transferable easily to other constructs, though. Seriously thanks, @kostix, for the great direction and time you've put here!
Answer: @kostix's answer suffices for most cases, but if you want to intercept tls.Conn.Handshake()'s reads and writes, you need to inject your mitmConn wrapper one level beneath.
Here's a working example:
package main

import "crypto/tls"

func dieIf(err error) {
    if err != nil {
        panic(err)
    }
}

type mitmConn struct {
    // Since we went one level beneath, no need for *tls.Conn here
    net.Conn
}

func (mc *mitmConn) Read(b []byte) (n int, err error) {
  return 0, nil
}

func (mc *mitmConn) Write(b []byte) (n int, err error) {
  return 0, nil
}

func main() {
    // Don't use tls.Listen
    l, err := net.Listen("tcp", "localhost:0")
    dieIf(err)
    c, err := l.Accept()
    dieIf(err)
    // Make a new tls.Conn.
    // After the tls.Server() call, the nesting of Conn
    // objects looks will be:
    // - *tls.Conn
    //  - mitmConn
    //   - *net.TCPConn
    c = tls.Server(mitmConn{c}, nil)
    // Since tls.Server returns a net.Conn, not a *tls.Conn, we need to cast
    err = c.(*tls.Conn).Handshake()
    dieIf(err)
}


Executing "SHOW PROCESSLIST" using PDO does not return the process list

Question: I am trying to get a list of the MySQL processes programmatically using PHP. But when I run this:
$dsn = "mysql:host=$host;dbname=$db;charset=$charset";
$options = [
    PDO::ATTR_ERRMODE            => PDO::ERRMODE_EXCEPTION,
    PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
    PDO::ATTR_EMULATE_PREPARES   => false,
];
try {
     $pdo = new PDO($dsn, $user, $pass, $options);
} catch (\PDOException $e) {
     throw new \PDOException($e->getMessage(), (int)$e->getCode());
}

$stmt = $pdo->query('SHOW PROCESSLIST');
while ($row = $stmt->fetch())
{
    print_r($row);
}

I get this output:
Array
(
    [Id] => 105818
    [User] => myuser
    [Host] => myfqdn.net:59158
    [db] => mydb
    [Command] => Execute
    [Time] => 0
    [State] => starting
    [Info] => SHOW PROCESSLIST
)

I have also tried this query:
$stmt = $pdo->query('SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST');
while ($row = $stmt->fetch())
{
    print_r($row);
}

With the same results:
Array
(
    [ID] => 105906
    [USER] => myuser
    [HOST] => myfqdn.net:59182
    [DB] => mydb
    [COMMAND] => Execute
    [TIME] => 0
    [STATE] => executing
    [INFO] => SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST
)

How do I actually get the real results from those queries, and why is this happening?
Comment: what else do you expect?
Answer: Your user probably doesn't have the PROCESS privilege, so it only has access to its own queries.
You need to run as root or user with grant option:
GRANT PROCESS ON *.* TO youruser@'yourhost';


Turn each array item into key value pair

Question: I have two arrays. One is a list of colors, the second is an associative array of key value pairs. My objective is to take the key value pairs from the associative array and make them a sub-array of each item in the colors array. Searching SO has gotten me various adjacent issues, but not the one I'm having specifically. Here are two example arrays, and then $final is what I want to achieve:
$colors = ['#eea845', '#64A0B4', '#003c50', '#FF5568', '#eee', '#5cb85c', '#5bc0de', '#f0ad4e', '#d9534f'];

$test = [
  'key1' => 'val1',
  'key2' => 'val2',
  'key3' => 'val3',
  'key4' => 'val4',
  'key5' => 'val5',
  'key6' => 'val6',
  'key7' => 'val7',
  'key8' => 'val8',
  'key9' => 'val9',
];

$final = [
   '#eea845' => [
      'name' => 'key1',
      'value' => 'val1',
   ],
   '#64A0B4' => [
      'name' => 'key2',
      'value' => 'val2',
   ],
   etc.....
]

I've been looking at array_walk, array_map, and trying to figure out how to combine for and foreach loops. I looked at the answer given here (Append one array values as key value pair to another array php) but I'm not sure how to use it on an already existing array and be able to get the index of each. For example, that solution uses array_walk($array1, function(&$v, $k) use($array2) { $v['date'] = $array2[$k]; }); but I need to have the values from $array2 be added to already existing items in $array1, and while I tried doing function($i, $v, $k) with $i being the index inside $array1, that didn't work, $i being undefined.
I'm stumped and not sure where to look next. How would you
return $colors[$i] =>
  [
   'name' => $test[$key],
   'value' => $test[$name]
  ]

?
(For context, I am using this to get values to input into a Twig template, and this looks like the best way to do it for that half of the problem. But if it's too difficult on this side...)
Comment: array_map then array_combine
Answer: I'd just use a combination of foreach() using the below logic:

Check if both the arrays have same length.
Use foreach to walk through the array and fill in the final array.
I am not good in this point, but I use a $count to keep track of the index. You can use cursors like next($colors) and current($colors), but I am not sure.

<?php
$colors = [
    '#eea845', '#64A0B4', '#003c50', '#FF5568', '#eee', '#5cb85c', '#5bc0de', '#f0ad4e', '#d9534f'
];

$test = [
  'key1' => 'val1',
  'key2' => 'val2',
  'key3' => 'val3',
  'key4' => 'val4',
  'key5' => 'val5',
  'key6' => 'val6',
  'key7' => 'val7',
  'key8' => 'val8',
  'key9' => 'val9',
];

if (count($colors) == count($test)) {
    $count = 0;
    $finalOne = array();
    foreach ($test as $key => $value) {
        $finalOne[$colors[$count]] = [
            "name" => $key,
            "value" => $value
        ];
        $count++;
    }
    print_r($finalOne);
} else {
    echo "Arrays are of not same length.";
}

Output
Array
(
    [#eea845] => Array
        (
            [name] => key1
            [value] => val1
        )

    [#64A0B4] => Array
        (
            [name] => key2
            [value] => val2
        )

    [#003c50] => Array
        (
            [name] => key3
            [value] => val3
        )

    [#FF5568] => Array
        (
            [name] => key4
            [value] => val4
        )

    [#eee] => Array
        (
            [name] => key5
            [value] => val5
        )

    [#5cb85c] => Array
        (
            [name] => key6
            [value] => val6
        )

    [#5bc0de] => Array
        (
            [name] => key7
            [value] => val7
        )

    [#f0ad4e] => Array
        (
            [name] => key8
            [value] => val8
        )

    [#d9534f] => Array
        (
            [name] => key9
            [value] => val9
        )

)

Working Demo: http://sandbox.onlinephpfunctions.com/code/73fb63fab8ccbdb6f73c795572ae51a0d98acdb4
Comment: Please use 3v4l.org as your demo'ing sandbox.  The UX on mobile devices is abysmal on sandbox.onlinephpfunctions.com
Comment: @mickmackusa This is the first time I am hearing about 3v4l.org. Thank you so much. ❤
Answer: There are several ways to do it. One way is to use foreach on the associative array, so you get key and value in separate variables, and to use next() to get the corresponding value from the first (indexed) array:
foreach ($test as $key => $value) {
    $final[current($colors)] = ["name" => $key, "value" => $value];
    next($colors);
}

In the rare event you had already used next() on $colors, you'll have to call reset($colors) before starting this loop.
Comment: Or with `array_shift()` if mutating/destroying the `$colors` array is tolerable. https://3v4l.org/G96iD  Or without destroying the `$colors` while maintaining a counter. https://3v4l.org/eMmUf


How do I find out what controls access to a page?

Question: I'm surprised by finding that I can't access [some-path] on my site, and I don't know what to do?
Where do I begin searching? What is the fundamental most level to start from?
Answer: For Drupal 6 and 7, the basic access check details are stored in the table menu_router.
To find out the relevant details, try a query like
SELECT path, access_callback, page_callback FROM menu_router WHERE path LIKE '%user%';
Lets look at three examples returned by that query on a local site of mine:
| path                | access_callback  | access_arguments   | page_callback
| user                | 1                | a:0:{}             | user_page
| user/%/edit/account | user_edit_access | a:1:{i:0;i:1;}     | drupal_get_form
| user/%/display      | user_access      | "administer users" | ds_contextual_page_tab

The data above has been manipulated slight for brevity.
First: user. userhas an access callback of  1, which means it will always pass. This is not, as one might think, to allow everyone access, but usually rather because the access decisions for this particular path are complicated enough to require more than a regular permission. Because of that, we need to look at the next column, which is page_callback. This tells us which function is called to actually handle the page, and page_arguments, in this case empty, could contain any arguments passed to the same function. In this case, there are no arguments, so we can't learn any more here. One needs to start reading the code of user_page() to figure out more.
Second: user/%/edit/account. This path has a standard access_callback, which means that user_edit_access is called, and if it returns TRUE, we'll be given access. One needs to read that function to find more details.
Third:  user/%/display. This one is the easiest. user_access simply checks if the user has been assigned the permission that is passed in from access_arguments, so in this case administer users. If that permission has been checked for the users role, access will be granted. User 1 automatically gets to bypass such checks.


Post hoc contrasts when only certain contrasts make sense

Question: My experiment has a 2x2 design, full between-subjects, so that the resulting four groups are as follows:
-Group A-1: Experimental treatment A.
-Group A-2: Control for experimental treatment A.
-Group B-1: Experimental treatment B.
-Group B-2: Control for experimental treatment B.
I am expecting that treatment A works (when compared to their control) while treatment B doesn't.
As the interaction is significant, I need to follow up with contrasts. In Jamovi and similar software, this is straightforward with Tukey's procedure.
However, note that not all the contrasts are useful here. In fact it makes no sense to compare the experimental treatment A with the control for the treatment B. Tukey just takes all possible combinations of the 4 means and correct accordingly.
Here only two comparisons are of interest: Treatment A vs. Control A, and Treatment B vs. Control B. Tukey corrects for 6 comparisons even if most of them are useless.
I've been reading about other procedures to protect alpha from multiple comparisons. Bonferroni is the easiest to apply but overly conservative (right?). Dunnett allows to reduce the number of contrasts but assumes only one control.
If I am interested in only two contrasts (and the interaction that reveals that treatment A works better than treatment B), which correction procedure can I apply?
Comment: Have you considered using [Holm's method](https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method)?
Comment: I don't think that would work. Holm-Bonferroni is a sequential method and still assumes that I'm interested in all 6 contrasts, whereas only 2 of them make sense.
Comment: Why would the method assume that you're interested in all 6 contrasts? It controls the FWER strongly within a prespecified "family" of to be tested null hypotheses.
Comment: Oh, perhaps this is something I don't understand. But this B-H procedure (as all sequential methods) will start by picking up the two most different means, or the smaller p-value, then proceed to the next, etc. If I followed this procedure, what if the first three or four constast that I test are completely useless because they are comparing the two controls with each other, or one treatment with the control from the other condition... And this leaves me exactly where I was when I asked the question: Do I need to assume that all combinations of groups need to be compared, and correct for them?
Comment: If you only want to test two *planned* contrasts, it is sufficient to apply Holm's method to the two corresponding hypothesis tests. But you might want to do three tests if your second hypothesis is about equivalence (see my answer).
Comment: You underline that you are interested in only two contrasts but actually you are doing three tests (two main effects and an interaction), so you have to control for three hypotheses rather than just two.
Answer: In what follows I assume that you want to control the FWER (in the strong sense). In general, if you want to test a fixed number of arbitrary planned contrasts (in your case: treatment A vs. control A, treatment B vs. control B), the Holm method can be used to control the FWER strongly within the family of hypothesis tests for these contrasts. The Holm method is more powerful than the Bonferroni correction, which could also be applied in this setting.
Note that this is to be distinguished from the case where contrasts suggested by the data (e.g., the difference between the two group means that differ the most) are tested. Here Scheffé's method (among other, more powerful, methods tailored to specific types of comparisons) could be used.
If the planned contrasts fully describe your hypotheses, there is also no need for an omnibus test.
Note also that your second hypothesis "treatment B doesn't work" suggests that you are looking for an equivalence test which, when based on two one-sided t-tests (TOST), would require rejecting two null hypotheses to support your (research) hypothesis.
Answer: This is a good question.
In your situation, it's common and perfectly ok to just run two t tests for the contrasts of interest, and do the Boneferroni correction manually by multiplying the p values by 2. With only two contrasts, there will be very little difference between this and other, harder to justify corrections.


jQuery Mobile : how exit an application on button click in native android application

Question: I have a HTML5+Jquery Mobile app, I converted the web view as Native android application using PhoneGap, I have written the following code to exit the application on Mobile. but this code not exiting the application. Please help me to resolve this issue.
jQuery :

HTML

        Back

        Next

Comment: I don't see anything with class ui-page-active and id home. Did you check if this condition was good ?
Comment: @Gaëtan " and ".ui-page-active" currently active page class (jquery mobile class)
Comment: @Gaëtan such classes are dynamically added by jQM once page is rendered.
Comment: if you add a console.log or alert, do they fire? **edit:** if yes, they will fire once as you bound `click` event to `paginit`.
Comment: its firing in browser, let me check in AVD
Comment: @Omar I'm getting console message on AVD, "11-22 12:07:30.922: I/chromium(1677): [INFO:CONSOLE(14)] "test", source: file:///android_asset/www/Index.html (14)"
Comment: @Omar in console log, on page finished event happen its going to page 5 "11-22 12:19:50.952: D/Cordova(1884): onPageFinished(file:///android_asset/www/Index.html#page5)
"
Comment: check this http://stackoverflow.com/questions/8602722/phonegap-android-back-button-close-app-with-back-button-on-homepage
Answer: At last I found what I missed, I should add the cordova.js on my applications javascript library.
Answer: Try change $('.ui-page-active').attr('id') == 'home' with $.mobile.activePage.is('#home')


One extreme outlier fitted value that is replaced by another when dropped

Question: I have an OLS model with a very bad prediction score - when I decided to test for heteroskedasticity, it turned out my model's predictions include one incredibe outlier - it looks like my fitted values plotted against residuals are all clustered in one cloud of dots, except there is one dot that has a way higher fitted value and residual, so high that the entire cloud of other dots has to be confined in the corner of the screen for this one dot to fit in the other. What's odd is that when i look that outlier up, it has no special, crazy qualities - very average X's across the board. Odder yet, if I manually drop it from my spreadsheet, another one takes its place, always in the row before or after the one I just dropped, with almost the same exact proportion of rows below to rows above. This is particularly puzzling, because I am using train-test split in python:
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)
which is supposed to shuffle values by default, so there is literally no significance to the position of the outliers. Why does it do that? How can it be remedied? The data, for those who want to replicate it, is here:
https://docs.google.com/spreadsheets/d/1Fa2fsgeRKy8Iodpt6FfDzf9oCxyNph5gIB-uD3T_ktQ/edit?usp=sharing
My entire code is here:
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
from statsmodels.stats.outliers_influence import variance_inflation_factor

pd.set_option("display.max_columns", None)
pd.set_option("display.max_rows", None)
# Read the data
df = pd.read_excel("final.xlsx")

# Assuming df is your DataFrame
y = df["rent"]
x = df.drop(
    [
        "rent",
        "Index",
        "pets_deposit",
        "Commute_nonrush",
        "Commute_rush",
        "school3",
        "walk-score",
        "bike-score",
        "lease_term",
        "transit-score",
        "cats_allowed",
        "pets_allowed",
        "deposit_missing",
        "application_fee_missing",
        "application_fee",
        "school2",
        "garbageincluded",
        "pets_allowed_deposit",
        "arrests",
    ],
    axis=1,
)

matrix = x.corr()
with open("OLS/matrix.txt", "w") as f:
    f.write(str(matrix))

# Calculate VIF for each variable
vif = pd.DataFrame()
vif["variables"] = x.columns
vif["VIF"] = [variance_inflation_factor(x.values, i) for i in range(x.shape[1])]
with open("OLS/vif.txt", "w") as f:
    f.write(str(vif))

# Split the data into training and testing sets
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)

# Perform the backward elimination
cols = list(x_train.columns)
p = []
x_1 = x_train[cols]
model = sm.OLS(y_train, x_1).fit()

# Calculate residuals
residuals = model.resid

# Find the index of the observation with the largest absolute residual
outlier_index = np.argmax(np.abs(residuals))

print(f"The index of the outlier is: {outlier_index}")

# If you want to see the entire row of the outlier in the DataFrame
print(df.iloc[outlier_index])

# print the prediction of the outlier
print(model.predict(x_1.iloc[outlier_index]))

# Create a scatterplot for residuals
plt.scatter(model.fittedvalues, residuals)
plt.xlabel("Fitted Values")
plt.ylabel("Residuals")
plt.title("Residuals vs Fitted Values")
plt.show()

selected_features_BE = cols
summary_str = model.summary().as_text()
with open("OLS/summary_OLS.txt", "w") as f:
    f.write(summary_str)

# Subset the test data to include only the selected features
x_test_BE = x_test[selected_features_BE]

# Make predictions on the test data
y_pred = model.predict(x_test_BE)

# Calculate and print the Mean Squared Error (MSE)
mse = mean_squared_error(y_test, y_pred)
print(mse)
with open("OLS/mse.txt", "w") as f:
    f.write(str(mse))

# Calculate and print the R-squared score
r2 = r2_score(y_test, y_pred)
print(r2)
with open("OLS/r2.txt", "w") as f:
    f.write(str(r2))
```

Answer: Are you running regression diagnostics to identify/drop overly influential observations, and then re-running with dropped outliers?  This entails looking at the Studentized residuals, jackknife residuals, Cook's distance, leverage residuals, DFFITS, DFBETAS, and comparing them with threshold values to identify and drop overly influential observations from your model.  VIFs can be used to look at multicollinearity between input predictors.
To sidestep regression diagnostics, you could try median regression (quantile regression), which will likely tone down the overly influential observations.  In addition, hopefully, you looked at univariate prediction models to get a handle on which predictors are meaningful by themselves.  Another way to "smooth out" these issues when looking for a relationship is to run univariate Somers' D between the x-y pairs, to get a handle on univariate directionality and association.
Comment: Thanks a lot - unfortunately, I am stuck with OLS - the topic of the thing I am writing is OLS specifically. I tried some of the things you are suggesting, not others, but I am not sure your answer applies - the issue, as I stated, is not that there is an outlier that has crazy X values that generate crazy Y values - not at all. In fact, the issue is not a datapoint at all. The issue is that an incredibly normal datapoint generates a fitted outlier, and that if I drop it, it is replaced with another outlier in the same row number in the database. That last part is very relevant.


Does the circle group given by drawing a parallel chord through the identity match the complex multiplicative unit circle group?

Question: I've been learning about homeomorphisms on the circle and was watching this excellent algebraic topology video covering some basics of projective geometry, and describing a group structure on the circle which is underpinned by Pascal's theorem.
The group product $X\star Y=Z$ is arrived at by drawing a chord through $X$ and $Y$ and then drawing a parallel chord through the identity, and where that chord meets the circle is the product $Z$.  It's not immediately obvious (at least to me) that drawing a unit circle in the complex plane and setting the group identity to be $1+0i$ makes $(S^1,\star)$ the standard multiplicative unit circle group in the complex plane $(S^1,\times)$.  But picking sample products such as $i\star i=-1$ and in fact any of the points of the compass, does yield matching results for $\star$ and $\times$.
I was able to satisfy myself that results matched for a good selection of elements of the Prufer 2-group, which being dense in the circle and would appear to order-embed, would determine this must be the same group.  Is it the exact same group and product?
Comment: So you want to see a proof that the "chord multplication" agrees with the standard multiplication?
Comment: @PaulFrost yes, I guess so.
Answer: Let $z_j = x_j + i y_j  \in S^1$ ($j = 1,2$) be two elements on the unit circle.
In order to show that the two multiplications are the same it suffices to show that the slope of the chord between $z_1$ and $z_2$ coincides with the slope of the chord between $z_1 z_2 = (x_1 x_2 - y_1 y_2) + i (x_1 y_2 + x_2 y_1)$ and $1 + 0i$.
That means we have to show that
$$ (x_1 - x_2) (x_1 y_2 + x_2 y_1) = (y_1 - y_2) (x_1 x_2 - y_1 y_2 - 1)$$
and with the help of $x_j^2 + y_j^2 = 1$ for $j = 1,2$, this is easily done.
Answer: I know this already has an answer, but there is a more geometric way of thinking about it that you might find nice.

The points $1, x, y, z$ form a cyclic quadrilateral. To avoid confusing notation, call these points $A, B, C, D$ respectively. Assume $x$ comes before $y$ as you traverse the circle anticlockwise.
Since the line $\vec{AD}$ is parallel to $\vec{BC}$, this quadrilateral is a trapezium.
Since it's a cyclic quadrilateral, $A+C = B+D = \pi$. Since it's a trapezium, $A+B = C+D = \pi$.
It follows the whole diagram is symmetric under reflection in the perpendicular bisector of AD (which is also the perpendicular bisector of BC).
If you add O, the center of the circle, to the diagram, then the desired fact to prove is

$$\arg x + \arg y = \arg z$$
i.e.
$$\angle AOB + \angle AOC = \angle AOD$$
But it's obvious (here we use the anticlockwise orientation of $ABCD$) that $\angle AOD = \angle AOC + \angle COD$, and by the symmetry $\angle COD = \angle AOB$. Hence,
$$\angle AOD = \angle AOC + \angle AOB$$
which is what we wanted to prove.
Comment: Note that I made some assumptions about how nicely the trapezium could be laid out. It's not obvious to me this proof covers the case where $\arg x + arg y > 2\pi$.
Answer: Too long for a comment.
Consider the reducible degree three curve $${\cal C}: z(x^2+y^2-z^2)=0$$ in ${\Bbb P}^2_{\Bbb R}$ with $O$ the point $(x:y:z)=(1:0:1).$
Let $A,B,C,$ and $O=(1,0)$ be points on $x^2+y^2-1=0$ in ${\Bbb R}^2={\Bbb A}^2_{z=1}\subset{\Bbb P}^2.$
$A$ and $B$ in ${\Bbb P}^2$ define a line that intersects ${\cal C}$ in a point I call $\infty_{AB}.$ Then proceed as with the elliptic curve group law:
$$A+B+\infty_{AB} =0$$ and similarly $$\infty_{OC} + O +C=0$$ remembering from projective geometry that $AB$ parallel to $OC$ means that $\infty_{AB}=\infty_{OC},$ we have $$A+B=-\infty_{AB}=-\infty_{OC}=O+C.$$ If we let $O$ be the identity for $+,$ we get $$A+B=C.$$


Is the sequence of functions uniformly convergent? Analyse the pointwise convergence.

Question: $n^2$$\frac{(1-cos(x/n))}{x}$
I don't know how to analyse the limit when n tends to infinity. Do you need to use Taylor series for that one?
Comment: Well, let's think about it pointwise first - we know that $n^{2} \to \infty$, and $x/n \to 0$, so $\cos(x/n) \to 1$, so $1-\cos(x/n) \to 0$. So what happens to the limit will depend on how fast $1-\cos(x/n) \to 0$; certainly a Taylor approximation will help!
Comment: I'm trying to use Taylor approximation on cos(x/n), but it's not very helpful.
Answer: Let $ f_n(x) = \displaystyle n^2\frac{1 - \cos(x/n)}{x} $
You have $ 1 - \cos(x/n) = \displaystyle \frac{x^2}{2n^2} - \frac{x^4}{24n^4} + o(x^4/n^4) $
So, by plugging in, you get that $ \displaystyle f_n(x) = \frac{x}{2} + o(x^2/n)$, and you get the pointwise convergence to $ x/2 $.
Now, notice that the limit function is unbounded near $ +\infty $, so you'll need to study the uniform convergence on compact subsets (closed intervals) of $ \mathbb{R} $, otherwise the $\infty$-norm won't exist.
On $ [a, b] $, we have that $ \displaystyle \|f_n - f \|_\infty = \|-\frac{x^3}{24n^2} + o(\frac{x^3}{n^2})\|_\infty \leq \frac{b^3}{24n^2} + o(\frac{1}{n^2}) \rightarrow 0 $, thus giving the uniform convergence, on compact subsets (which is enough for all limit switching theorems).


Creating a custom field type

Question: I'm trying to create a custom FieldType which provides the fields for a complete address (street, house_number, post_code and city).
I created a FieldWidget, FieldFormatter, and a FieldType class and a field template, following Drupal 8: Creating a custom field - Part 1: Field type.
The widget is not rendered, unless I add a hook_element_info() implementation and a hook_preprocess_address() where I have to define the same stuff defined in FieldWidget::formElement().
This is the Drupal 7 way, though. Am I missing something?
FieldWidget class

public function formElement(FieldItemListInterface $items, $delta, array $element, array &$form, FormStateInterface $form_state) {
  $element = $element + array(
    'street' => array(
      '#title' => $this->t('Street'),
      '#type' => 'textfield',
      '#default_value' => isset($items[$delta]->street) ? $items[$delta]->street : NULL,
      '#required' => FALSE,
    ),
    'street_nr' => array( /* … */),
    'zip' => array( /* … */),
    'city' => array( /* … */),
  );

  $form['#validate'][] = array($this, 'validateCompleteAddress');

  return $element;
}

FieldFormater class
public function viewElements(FieldItemListInterface $items) {
  $elements = array();

  $elements['addresses'] = array(
    '#type' => 'fieldset',
    '#tree' => true,
  );

  foreach ($items as $delta => $item) {
    $elements['addresses'][$delta] = array(
      '#theme' => 'fields_address',
      'elements' => $item,
      // 'street' => String::checkPlain($item->street),
      // 'street_nr' => String::checkPlain($item->street_nr),
      // 'zip' => String::checkPlain($item->zip),
      // 'city' => String::checkPlain($item->city),
    );
  }

FieldType class
public static function propertyDefinitions(FieldStorageDefinitionInterface $field_definition) {
  $properties = array(
    'street' => DataDefinition::create('string')->setLabel(t('Street'))->addConstraint('Length', array('max' => 255)),
    'street_nr' => DataDefinition::create('string')->setLabel(t('Street number'))->addConstraint('Length', array('max' => 100)),
    'zip' => DataDefinition::create('string')->setLabel(t('Zip'))->addConstraint('Length', array('max' => 5)),
    'city' => DataDefinition::create('string')->setLabel(t('City'))->addConstraint('Length', array('max' => 255)),
  );

  return $properties;
}

public static function schema(FieldStorageDefinitionInterface $field_definition) {
  return array(
    'columns' => array(
      'street' => array(/* … */),
      'street_nr' => array(/* … */),
      'zip' => array(/* … */),
      'city' => array(/* … */),
     ),
     'indexes' => array(),
     'foreign keys' => array(),
   );
}

Comment: It is not clear what exactly you are asking. There is *nothing* special about what viewElements() returns and what you can do anywhere else with render arrays. If you use theme functions or special #type's, then you need to define them yourself. There is no magic.
Comment: This question appears to be off-topic because it is about code you wrote, for which you want to write better code, or you want to understand why it doesn't work, but you didn't show the code you are using, or the part of the code that is relevant for the question. [See Help Center](http://drupal.stackexchange.com/help/on-topic).
Comment: added some code of the classes
Comment: It was not clear to me, that one has to define the same stuff **twice**. (backend and frontend)
Comment: Seems like one has to provide a custom "Element" class in module\Element folder to define custom Input elements for the form rendering:

      @see \Drupal\Core\Render\Element\Textfield
Answer: It seems like I have to provide a custom Element class in the module\Element folder to define the custom input elements for the form rendering.
See \Drupal\Core\Render\Element\Textfield.

Creating a custom field type


Ant depends attribute leading to errant behavior

Question: I have two Ant tasks. If I run them, one after the other, from the command line -- no problems.
But if I include them in the depends attribute of another task like this depends='taskA,taskB' I get errant results.
Are the depends being run in series instead of in sequence?  Without going into the specifics of the ant tasks -- any ideas?
Comment: They are run in sequence.  To get a useful answer, you are going to have to share more about what isn't happening as expected.  How about adding an echo to your three Ant tasks and commenting on the order/output?
Answer: ANT only executes dependent tasks once... The example below illustrates this strange behaviour.
This falls into the category of feature rather than bug. It's just the way ANT works. The thing to bear in mind is that ANT is not designed as a procedural programming language.
If your sub task is procedural in nature it would be best invoked as a macrodef (or better again create a re-usable antlib of build functions).
Example
$ ant
Buildfile: /path/to/my/home/build.xml

subtask1:
     [echo] hello world

subtask2:
     [echo] hello world

build1:
     [echo] hello world

build2:
     [echo] hello world

build:

BUILD SUCCESSFUL

build.xml
<project name="demo" default="build">

   <target name="build" depends="build1,build2"/>

   <target name="build1" depends="subtask1,subtask2">
      <echo message="hello world"/>
   </target>

   <target name="build2" depends="subtask1,subtask2">
      <echo message="hello world"/>
   </target>

   <target name="subtask1">
      <echo message="hello world"/>
   </target>

   <target name="subtask2">
      <echo message="hello world"/>
   </target>

</project>


Divergence of improper integral $\int_{2}^{\infty} \frac{\cos(2x)\cos(6x)}{x\ln x}$

Question: Given the integral $$\int_{2}^{\infty} \frac{\cos(2x)\cos(6x)}{x\ln x},$$ is there an easier way to show its divergence than starting to break this integral to a sum of smaller integrals using trigonometric identities?
Can I check the limit  $\lim_{x\rightarrow \infty}\frac{\cos(2x)\cos(6x)}{x\ln x}*x\ln x$ and given that $\frac{1}{x\ln x}$ is diverging say that  $\int_{2}^{\infty} \frac{\cos(2x)\cos(6x)}{x\ln x}$ is diverging?
Comment: integral by parts can help?
Comment: The integral is converging, and the mentioned limit does not even exist.
Answer: Both $\cos(4x)$ and $\cos(8x)$ are functions with a bounded primitive, and the function $\frac{1}{x\log x}$ is continuous, decreasing and converging to zero on $[2,+\infty)$, hence the integral
$$ \int_{2}^{+\infty}\frac{\cos(2x)\cos(6x)}{x\log x}\,dx $$
is convergent by Dirichlet's test. Numerically,

$$\int_{2}^{+\infty}\frac{\cos(2x)\cos(6x)}{x\log x}\,dx = -0.0762839\ldots $$

On the other hand, the integral of the absolute value is (very slowly) diverging, since:
$$\int_{e}^{N}\frac{\left|\cos(2x)\cos(6x)\right|}{x\log x}= \frac{3\sqrt{3}}{4\pi}\log\log N +O\left(\frac{1}{N\log N}\right)$$
by the same reason.
Comment: Why does it diverging? Did you calculated the value of the integral in absolute value and concluded that it's diverging?
Comment: @YinonEliraz: the last estimate follows by integration by parts and proves that the integral of the absolute value is diverging since $\log\log N$ is unbounded as $N\to +\infty$.


PHPExcel: long numbers have scientific format (xxxE-08) when reading

Question: When reading an Excel file with PHPExcel Reader, long numbers are printed out in scientific format, so 0.0000000123 becomes 1.23E-08.
The documentation says I can avoid this with Advanced Value Binder, but unfortunately this only works with CSV files.
Also setReadDataOnly() does not help.
It also happens if the cell is formatted with long floating numbers so it is displayed as 0.0000000123 in Excel.
There are some topics here for writing, but not for reading.
Question: How do I fetch the number w/o scientific format?
Answer: Just came across the idea, that PHP is responsible for echo scientific notation, so I have tried number_format($var, 8); and it worked.
So it's not PHPExcel. I have to echo using number_format.


Sorting list by alphabetical order, with special case for certain starting letter

Question: Given a list of strings, I want to return a list with the strings in sorted order, except group all the strings that begin with 'x' first.
For example:
['mix', 'xyz', 'apple', 'xanadu', 'aardvark']

Should yield:
['xanadu', 'xyz', 'aardvark', 'apple', 'mix']

I know it can be done by making 2 lists and sorting each of them before combining them, but I'm not sure how to write the condition in which 'x' is the first character.
Comment: and what did you tried to code?
Comment: rsm - I'm actually pretty new at this, so I didn't know what to do. I searched this and other sites for a good answer, but couldn't find any, so I posted this here, hoping to get some assistance. Thanks for all your answers, people! (and to Will, for editing my question).
Answer: sorted or list.sort accepts optional key keyword argument. That is the function used to get sort key, and the return value of the function is used to compare instead of the original items.
>>> words = ['mix', 'xyz', 'apple', 'xanadu', 'aardvark']
>>> sorted(words, key=lambda word: (word[0] != 'x', word))
['xanadu', 'xyz', 'aardvark', 'apple', 'mix']

used word[0] != 'x'; which return False (0) for word starts with x, True (1) for other words; resulting words start with x come first.
Answer: Just make a particular case in your key method: if starts with x, return a truncated string starting with "0" so it will appear first, but will be still sorted after "x".
z=['mix', 'xyz', 'apple', 'xanadu', 'aardvark']

z.sort(key=lambda x : "0"+x if x.startswith("x") else x)
print(z)

yields:
['xanadu', 'xyz', 'aardvark', 'apple', 'mix']

Answer: words = ['mix', 'xyz', 'apple', 'xanadu', 'aardvark']
result = [i for _, i in sorted((word[0]!='x', word) for word in words)]

Comment: Hi, could you please add some explanation to your code?
Comment: using `_` as a variable name is strange for sure. I didn't even know that it worked. Cryptic as hell! Please explain that the returned key is a tuple, and sorting on tuples works by trying first value, and so on which explains why it works.
Answer: >>> sorted(['mix', 'xyz', 'apple', 'xanadu', 'aardvark'],
           key=lambda x: (not x.startswith('x'), x))
['xanadu', 'xyz', 'aardvark', 'apple', 'mix']

The sorted() builtin returns a new, stable-sorted list() of each element in the input iterable, sorted by key. key, in this case, is a lambda expression (basically a "mini-function") which transforms or converts each element in the input list into a sortable value.
In this case, our lambda expression sorts each word in the list by a tuple() which contains False or True indicating whether or not the word starts with "x", followed by the word itself. Since False is "smaller" than True, the words that start with "x" appear first, and everything else is sorted alphabetically.
Comment: @Will Well, I'm sure you could've explained in even much more detail, but you didn't, right? You stopped at some level, just like I did. Our levels just differ. And it's not like I don't want to explain anything, I also did write a [fairly long answer](http://stackoverflow.com/a/38164742/1672429) the same day. But don't worry, the reason I'm trying to stay away is just that I have much more important stuff that I should be doing instead.
Comment: Hi, could you please add some explanation to your code? This popped up in the review queue, as code-only answers tend to.
Comment: @Will I believe it's self-explanatory. You don't?
Comment: The code does speak for itself to me, as an experienced Python engineer. However, new users who stumble across this may not understand `lambdas`, `sorted()` or `str.startswith()`. Additionally, adding plain-English explanation is just a part of SO's culture (not sure if an actual requirement). I'll add the type of brief explanation I'm talking about as an example; feel free to improve.
Comment: Ok, check out my edit :) Remember, your audience doesn't always know what you or I know and take for granted.
Comment: @Will I guess it might help some people, but I wouldn't have the patience to preemptively explain such basic stuff. I'd say *"my audience"* is the people who couldn't or wouldn't come up with my code, but *can* understand it, at least after playing with it or looking stuff up. But ok, I'll just try harder to stay away from SO.
Comment: Seems to be a bit of a toxic attitude. Regardless, code-only answers pop up in the review queue, so you can just include a single sentence of explanation (that isn't "Try this:") and they won't pop up in the review queue. From your stats, it looks like you've helped a lot of people here and it would be a shame to see you spend less time on SO. I never voted to delete your post, and in fact upvoted it. My suggestions are only meant as constructive criticism, and you certainly don't need to add as much explanation as I did. Please don't let my suggestions make you not want to be here. I'm sorry.
Comment: Right, the only takeaway here is that if it doesn't have a sentence of text, it gets popped in the review queue. Your other answer is excellent, upvoted that as well :)
Comment: Here's an example of [one of my very short ones](http://stackoverflow.com/a/37711234/145279) that is mostly code-only. So you're absolutely right; you don't need significant explanation in many cases.
Answer: You can sort it using the built-in list.sort() and then use list comprehension to get the output you desire as follows:
sl = ['mix', 'xyz', 'apple', 'xanadu', 'aardvark']
sl.sort()
sl = [el for el in sl if el.startswith("x")]+[el for el in sl if not el.startswith("x")]


Convergence of Series of Functions

Question: Let $f(x)= \displaystyle \sum \limits_{n=1}^\infty \frac{\sin(nx)}{n^3}.$ Show that $f(x)$ is differentiable and that the derivative $f'(x)$ is continuous.
In class we solved a similar problem, and I think we had to show that both $f(x)$ and $f'(x)$ converge uniformly, but I am not really sure why that is what we have to show. I think we also showed that the partial sums are continuous.
I would really appreciate it if someone could solve this problem and maybe explain a little the theory behind it.
Thanks!
Comment: Just as an aside, this is a Clausen function: http://mathworld.wolfram.com/ClausenFunction.html
Answer: To show uniform convergence of the original series and the series of derivatives you can use the facts that $\sin$ and $\cos$ are bounded and that $\sum_{n=1}^\infty \frac{1}{n^3}$ and $\sum_{n=1}^\infty \frac{1}{n^2}$ converge.
The uniform limit of continuous functions is continuous.  To see that $f$ is differentiable with derivative equal to the series of derivatives, you can apply an integral convergence theorem.  E.g., uniform convergence allows you to pass limits through integrals, and (using the fundamental theorem of calculus) $f(x)=\sum_{n=1}^\infty \int_0^x \frac{\cos(nt)}{n^2}dt$.  You can use the fundamental theorem of calculus again to finish, also using the fact that the series of derivatives represents a continuous function by the first sentence of this paragraph.
(I'm intentionally leaving some details for you to fill in.)
Comment: Nicely given. Let me just remark that in your final sentence one can drop the condition of continuity of the derivative. The reference is theorem 7.17 of Rudin's Principles of Mathematical Analysis, 3rd Ed., and the proof is a little bit longer.
Comment: Is there a way of proving differentiability without using the integral convergence theorem? We haven't learned this theorem yet.
Comment: @Daniel: The lemma you could prove and apply here is that if $(g_n)$ is a sequence of continuous functions converging uniformly to $g$ on the interval $[a,b]$, then $\int_a^b g(t)dt=\lim_{n\to\infty}\int_a^b g_n(t)dt$.  It is pretty straightforward.  If $|g(t)-g_n(t)|\lt\varepsilon$ for all $t$, then what can you say about $|\int_a^b g(t)-g_n(t)dt|$?  In the application to your problem, $(g_n)$ will be a sequence of partial sums.
Comment: @George: [clarification of uneditable comment I'm deleting.] Yes, continuity allows the integrability of the series of derivatives to follow, so that the FTC can be applied, and without knowing this a priori it takes considerably more work. (And in general, the derivative need not even be Riemann integrable.  E.g., one could consider a constant sequence consisting of an everywhere differentiable function without integrable derivative, as seen e.g. in http://mathoverflow.net/questions/6711/integrability-of-derivatives/6716#6716.) Thanks for the comment.
Answer: You had to show that both $(f_n)$ and $(f_n')$ converge uniformly because you want to apply a theorem like this:
Theorem 10.7 (from "Real Analysis" by N.L. Carothers):
Suppose that $(f_n)$ is a sequence of real-valuled functions, each having a continuous derivative on $[a,b]$, and suppose that the sequence of derivatives $(f_n')$ converges uniformly to a function $g$ on $[a,b]$. If $(f_n(x_0))$ converges at any point $x_0$ in $[a,b]$, then in fact, $(f_n)$ converges uniformly to a differentiable function $f$ on $[a,b]$. Moreover $f' = g$. That is, $(f_n')$ converges uniformly to $f'$ on $[a,b]$.
Answer: Well, to see that $f(x)$ *converges uniformly* please apply the Weierstrass M-Test.
Now, i am aware of the fact that term by term differentiation and integration is possible if the series is uniformly convergent. But i don't know much about this, i don't want to answer that, but i am sure someone here will help you out regarding this.
Comment: To differentiate term by term you need the series of derivatives to also converge uniformly, and this is probably what you meant.  A uniform limit of differentiable functions need not be differentiable (e.g., see the Weierstrass approximation theorem and the existence of continuous nowhere differentiable functions).
Comment: (For a lower tech example, consider the sequence of functions $g_n(x)=\frac{\sin(nx)}{n}$, which converges uniformly to $0$ but whose sequence of derivatives does not converge.)
Comment: @jonas: yes u are right!

Convergence of Series of Functions


Hibernate: Join 2 tables on a property

Question: I have 2 tables with the following properties in the prod DB:
Bills table : id, customer_id, total
Project table : id, customer_id

and customer_id is not a foreign key in any of the tables. Is there a way to join these tables with hibernate criteria? Tried using DetachedCriteriawith Subqueries criteria. But was not able to join the tables!
Answer: You would have to change your mapping so that you have an additional mapping in your Bills:
@ManyToOne
@JoinColumn(name = "customer_id", referencedColumnName = "customer_id")
private Project project;

Then you can use the standard way of joining in the criteria.
Update
If you can use HQL instead of Criteria.. then you can use the old style of join:
select b.name
from Bills b, Project p
where b.customer_id = p.customer_id

Update 2
If you have to use xml then specify your  as:
<many-to-one name="project" class="com.Project" ..>
   <column name="customer_id" property-ref="customer_id"/>
</many-to-one>

Comment: Thanks for the reply Maciej, but unfortunately I cant edit the mappings and we are not using annotation based mappings. Is there any other way?
Comment: I guess I can. "". This is what I have in my bills hibernate.xml where Client is a different table
Comment: I have updated the post if you can use hql instead of criteria. The benefit would be that you can make it a NamedQuery and speed up the overall query time.
Comment: not really, but got the DetachedCriteria working. Thanks for the concern

Hibernate: Join 2 tables on a property


Question about how auto.arima works

Question: I have a time series that grows by 10% every month; I have done auto.arima on the month-over-month growth rate and absolute monthly numbers; for example

Absolute numbers = 100, 110, 121, 133.1, ...
Month-on-month-growth = 0.1, 0.1, 0.1, ...

When I auto.arima on the growth rate as expected it gives me a forecast of the next month to be another 0.1, but when I do it on absolute numbers I get a implied growth of 0.0909.
Can someone please explain why there is a difference?
library(forecast)
rates = ts(rep(0.1,48), frequency = 12, start = c(1959, 1))
auto.arima(rates)
f  = auto.arima(rates)
print(forecast(f, 1))

foo = 1:48 ;
base = 1;

for ( i in 1:48 ) {
        foo[i] = base;
        base = base * 1.1;
}

levels = ts(foo,frequency = 12, start = c(1959,1))
fl = auto.arima(levels)
print(forecast(fl,1))

Comment: What is the form of the model that the software picked?  Was it the same ib both cases?
Answer: Your first series is generated exactly by a (degenerate) ARIMA process: it's ARIMA(0,0,0), with a non-zero mean (0.1), and zero variance. auto.arima is able to find this model correctly, and since it is degenerate it can easily forecast the next values exactly.
Your second series is not generated by a nice (stable) ARIMA process (it's a degenerate AR(1) with $\phi > 1$). auto.arima will try to find the best model (according to AICc) within the universe of nice ARIMA models, but will not be able to find one that fits it exactly (it settles on ARIMA(0,2,0) with no mean).
You can fit your second series into the ARIMA framework by using a transformation, specifically a log-transform, such as by setting lambda = 0 in the call to auto.arima (auto.arima does not try to set lambda automatically for you). In that case, it will find ARIMA(0,1,0) as the best model, but without drift, which your series does have. The reason for this is presumably because including the drift throws an error (system is exactly singular), which is caused by the fact that the "data" you are using isn't really data and does not contain any variability at all.

Question about how auto.arima works


Graphviz2 in perl to save graph image

Question: I would like to output the graph generated by using GraphViz2 perl module to a file. I have the next code, but it doesn't save the image. How can I save it?
my $output_dir = "/home/user/Desktop/output.png";

my($graph) = GraphViz2 -> new ();

                (
                 edge   => {color => 'grey'},
                 global => {directed => 1},
                 graph  => {label => 'Adult', rankdir => 'TB'},
                 logger => $logger,
                 node   => {shape => 'oval'},
                );

        $graph -> add_node(name => 'Carnegie', shape => 'circle');
        $graph -> add_node(name => 'Murrumbeena', shape => 'box', color => 'green');
        $graph -> add_node(name => 'Oakleigh',    color => 'blue');

        $graph -> add_edge(from => 'Murrumbeena', to    => 'Carnegie', arrowsize => 2);
        $graph -> add_edge(from => 'Murrumbeena', to    => 'Oakleigh', color => 'brown');

my($format)      = shift || 'png';
my($output_dir) = shift || File::Spec -> catfile('png', "sub.graph.$format");

$graph -> run(format => $format, output_file => $output_file);

Answer: my($output_dir) = shift || File::Spec -> catfile('png', "sub.graph.$format");

should be
my($output_file) = shift || File::Spec -> catfile('html', "sub.graph.$format");

Your code has no $output_file you've made a typo or copied/pasted code incorrectly from somewhere.

Graphviz2 in perl to save graph image


Using a single view for different data in android

Question: let's say that we have one edit text and one button and a single text view in an activity.
If each time I type data into the edit text and hit the button I want to get that entered data independently, example(If I type cat I want to see a cat text on the activity/ then when I enter dog I want to see also dog on the activity), the result is a ''cat text'' and beside it a ''dog text'' .
The problem : how can that be achieved using only a single text view for both data.
Answer: onClick of button write this code:
textView.setText(textView.getText().toString()+ " " + editText.getText().toString());

Comment: Hey @BhaveshMisri that's not what I meant , What I need is to reuse the textview for multiple data, each data is independent of the other but all of them use the same view.
Comment: @HasanB.T Data should be independent but view should be same? That's not possible; what you can do is create a new TextView at runtime.
Comment: I dont want to add 6 textviews for 6 entered texts, I want to add into the activity only one textview for all possible values entered.
Comment: I guess it's a recyclerview thing but I dont know how it can be accomplished.
Comment: @HasanB.T yea, you can do it by recyclerView too. You haven't used it before?
Comment: I know how to implement it but I dont know how to set it in framelayout so that each view is above the other on Z-axis. And I dont know in what method I can get the text entered in edit text.
Answer: Instead of using
textView.setText("cat text");

use
textView.append("cat text");

it will appends the text
to goto next line then,
textView.append("cat text\n");

it will result in,
cat text
dog text
TextView can be made programaticaly
TextView textView = new TextView(this);
this will create a Textview Programatically.And also you can edit the properties like,
textView.setText("bla bla bla");
textView.setTextSize(...
textView.setTextColor(...

Comment: if let's say I want to cause (cat text ) to be dragged on touch and then I want the (dog text) to be dragged on touch, but both are independent of each other so that cat text can't be dragged when dog text is dragged.How would that be achieved.
Comment: generate the TextViews programatically.
Comment: you mean that Views (imageview, Text view.........) can be added to the activity through code.


In UWP app, where should html assets be stored in the assets directory?

Question: In a UWP app using cppwinrt I want to use WebView to display contents of a book kept in the Assets folder. I read that it is necessary to reference an html asset this way for use as a Uri argument to the Navigate method in web view:
TheWebView.Navigate(Uri(L"ms-appx-web:///SampleBook/PageOne.html"));

This produces an empty view, while
TheWebView.Navigate(Uri(L"ms-appx:///SampleBook/PageOne.html"));

crashes. Msdn says that for files "that will be loaded into the web compartment" one must use ms-appx-web, and I've seen mention that this is a security issue. But does that mean the files are in a special location within the project - i.e. not merely in the Assets folder - or does it only mean that the path must begin with ms-appx-web independent of the file's location? "Web compartment" is not explained but seems to be not a location but rather a classification of the type of resource. At any rate, neither of the above approaches works, so I'm curious to know the recommended way to store and access a collection of html files in the package. In the assets folder? A special folder within assets? In Solution Explorer the html file is listed, "content" is True, and the file is Included In Project. Thanks.
Comment: Did you figure this out? I'm trying to figure out the recommended way as well.
Comment: ShrimpCrackers, sorry, no - I haven't gotten back to that part of the project yet and no one has offered a suggestion. Seems to me the answer might have to do with finding an answer to the question: what is a 'web compartment' -?
Comment: ShrimpCrackers, I got it. Silly error on my part: the path was wrong. I have the folder with html content in the Assets directory, but that ms-appx-web does not go to assets - the required path was ms-appx-web///Assets/SampleBook/PageOne.html. The term "web compartment" in the docs has no meaning I've been able to discover - seems to refer just to any kind html content. Now I just need to find out why the videos in my content won't play.
Answer: My mistake: ms-appx-web does not point to the assets folder, but to its parent. The correct path for content of this type would be ms-appx-web///Assets/SampleBook/PageOne.html. The reference to material to be "loaded to the web compartment" apparently is just a way of saying: stuff to be loaded with WebViewer.


Show full list of nodes that reference the node being shown

Question: I have a content type (let's call it course) and a content type named edition (if we have a course named "English 101", we can have "English 101 - June", "English 101 - July" and so on.) For this, I used Entity Reference.
If I see the node "English 101-June", I can see the "parent" node being rendered (I chose that on Manage Display". What I want now, is when I see the "parent" node "English 101" the full list of referenced nodes to be rendered. How can I do this?
Answer: There are a number of ways to get reverse references. Probably the best option is to use the Views module. The reference module allows to create a reverse relationship in the view. You could do something like this:

With the Views module, create a view of nodes with a page display at path 'myview' ;
Set up your view to display only the title of each node ;
In your view add an argument that matches your node id ;
In your view add a reverse relationship that matches the field that is used to reference content. The name to use for the relationship would be something like field_my_reference_field (reverse)

Save the view, now go to 'myview/1' and you will see all the nodes that reference node 1. You could also get this displayed in a block, or as a field using the ViewField module.
If you don't want to use the Views module, then you can use the Nodereferrer module to achieve this. However in that case you can only get the list as a field - you can't get it as page or a block.
Answer: You can create a first view to list all your courses nodes.
And next, in a custom module, you display the references  (with antoher view who accept an argument of the parent reference nid)
function custommodule_entity_view($entity, $type, $view_mode, $langcode) {

  list($id, $rev, $bundle) = entity_extract_ids($type, $entity);
  switch ($bundle){
  case 'course':
    if( $view_mode == 'full' ){
      $entity->children['children'] = array(
        '#markup' => '
Childrens
'. views_embed_view( 'childrens', 'default', $entity->field_course['und'][0]['nid'] ),
      );
    }
    break;
  }
}

Comment: Maybe I didn't explain well. What I want is to list the nodes when I am viewing the "parent" node, not a list with all nodes and the references
Comment: you can add a condition to display only if the node is in full page : if( preg_match('#^node/.*#', $_GET['q'], $matches) );


Robust Sparse Clustering: how do we choose L1 and alpha parameters?

Question: Before asking, I would like to say that I was not sure where to ask this question - on CrossValidated or on StackOverflow. Finally, it seems to me more logical to ask it here. So, I've read an article (Tibshirani & Witten, 2010) on Robust Sparse Clustering and wanted to perform this algorithm on a sample data. However, I did not get the thing about L1 weights. What principle lies behind the choosing the boundaries for feature weights?
I may be misunderstood that is why I will provide some sample R code:
test <- matrix(runif(500), nrow = 500, ncol = 45)
RSKC::RSKC(test, ncl = 6, alpha = ?, L1 = ?, ...)

or using the other package for robust sparse clustering:
sparcl::KMeansSparseCluster(test, K = 6, wbounds = ?)

How do we decide the alpha and the L1 when clustering data with lots of features? I would appreciate the answer that will include simple explanation.
Answer: The sparcl has the KMeansSparseCluster.permute function for choosing the wbounds, see its help page for an example how to use this.
If I understand their paper well (where this is called $s$), the idea is the following. In principle they want to optimise the weighted k-means objective function. If the $s$ increases, the objective function will get better, but they have less sparsity. They want a good value of the objective function but still sparsity, so they've got to find a compromise. Now in order to assess the quality of a value of $s$ for finding an "optimum", they compute the solution for the given $s$, take a logarithm of the objective function, and compare it with the average logarithm of the objective function that they get when randomly and independently permuting the values on all variables (the logarithm is chosen because it scales better over different values of $s$; the plain objective function is bounded by zero, and values close to zero - and their differences - don't compare well with larger values). They then select $s$ so that the difference between these two is maximised (there is also an alternative suggestion to choose the smallest $s$ that is within one standard deviation of the best value; in fact I think one could even choose two standard deviations there, as the logic comes from the gap statistic, where personal experiments led me to believe that using 2 standard errors is probably good).
The rationale is that the permuted data sets are meant to give information about how log of the objective function is expected to behave if there is no meaningful clustering structure in the data. If an $s$ under investigation achieves a much better value on the original data set than on the permuted versions, this indicates that this $s$ is connected to some meaningful structure.
You may ask: Why are permutations of the original data suitable for this? Indeed there may be problems with this. If the values of one variable are strongly clustered on their own, they will still be strongly clustered in the permuted version. If there are not many variables, this may still bring forth a meaningful clustering, so permutation does not necessarily remove all clustering structure. The idea is probably (this isn't discussed in much detail in the paper) that if clustering structure in the original data is supported by a number of correlated variables, the permutation will destroy that and therefore serves as a benchmark. If an $s$ does much better on the original data, it would indicate that such structure is successfully preserved with that $s$. The authors also write: "the performance of the gap statistic of Section 3.2 for selecting the tuning parameter is mixed. This is not surprising, since tuning parameter selection in the unsupervised setting is known to be a very difficult problem. This is an area in which more work is needed." I'm not aware of anybody who has done this work though.
PS: I haven't read Kondo's paper on which RSKC is based, but it seems that the L1 parameter corresponds to the $s$ above. The alpha parameter is a trimming rate, which does something different. This is discussed in the robust clustering literature and there is no automatic way to choose it. People in robust clustering will plot the objective function against various choices of alpha to see whether a certain value sticks out. A shortcut is to choose alpha as the biggest proportion of data that if observed close together would not count as "cluster" but rather as a "bunch of outliers", or in other words a little bit smaller than the smallest size (proportion) for a cluster to be meaningful, in order to avoid that complete meaningful clusters are trimmed. This has to be decided on subject matter grounds. Unfortunately, unless you choose alpha=0, the KMeansSparseCluster.permute function won't work to choose L1.
Comment: Speaking generally, the main purpose of unsupervised learning is to reduce the dimensionality of features that are passed on to supervised learning.  Supervised learning has its limitations (with regard to overfitting) from the effective sample size of what's being predicted.  So one important consideration in choosing the number of principal components is this effective sample size, which needs to somehow be factored into the calculation.


How to create input data for a recurrent network in CNTK (C#)?

Question: I need to create data for a mini-batch with shape: (5(input_vector), 50(sequence_length), 100(amount_of_input_sequences)), in order to run 100 instances of input data for a recurrent neural network in 1 run of model.Evaluate.
At the moment I am unable to create input data even for 1 run. My code produces the output for 50 runs of the neural network instead of using the data as a single sequence.
I did not find an example of creating input data for RNN using the Value object. I also did not find an example of creating an input tensor with three axes.
I would be glad if someone tells me how to do this.
`var cpu_device = DeviceDescriptor.CPUDevice;
var gpu_device = DeviceDescriptor.GPUDevice(0);
var model_device = gpu_device;

int inputDim = 5;
int cellDim = 5;
int outputDim = 3;
int sequenceLength = 50;
int inputSequences = 100;

NDShape inputShape = NDShape.CreateNDShape(new int[] { inputDim });
NDShape outputShape = NDShape.CreateNDShape(new int[] { outputDim });

var inputVariable = Variable.InputVariable(inputShape, DataType.Float);
var outputVariable = Variable.InputVariable(outputShape, DataType.Float);

var lstmLayer = CntkWrapper.Layers.LSTM<float>(cellDim, inputVariable, model_device);
var model = CntkWrapper.Layers.Dense<float>(outputDim, lstmLayer, CNTKLib.Sigmoid, model_device);

Random random = new Random();
var inputs = new float[inputSequences][][];
for(int i = 0; i < inputs.Length; i++)
{
    inputs[i] = new float[sequenceLength][];
    for(int k = 0; k < inputs[i].Length; k++)
    {
        inputs[i][k] = new float[inputDim];
        for(int p = 0; p < inputs[i][k].Length; p++)
        {
            inputs[i][k][p] = (float)random.NextDouble();
        }
    }
}

NDArrayView[] sequenceNDArrayView = inputs[0].Select(o => new NDArrayView(inputShape, o, cpu_device)).ToArray();
var gpuInputsValue = Value.Create(inputShape, sequenceNDArrayView, gpu_device);

var inputDataMap = new Dictionary<Variable, Value>() { { inputVariable, gpuInputsValue } };
var outputDataMap = new Dictionary<Variable, Value>() { { model.Output, null } };
model.Evaluate(inputDataMap, outputDataMap, gpu_device);
var outputValue = outputDataMap[model.Output];
IList<IList<float>> actualLabelSoftMax = outputValue.GetDenseData<float>(model.Output);`

`public static class Layers
{
    public static uint Seed = 1;
    public static double NormalMean = 0.0;
    public static double StandartDev = 0.25;

    public static Function Dense<ElementType>(int dim, Variable previousLayer, Func<Variable, Function> activationFunction, DeviceDescriptor device)
    {
        int inputDimension = previousLayer.Shape.Dimensions[0];
        var weights = new Parameter(NDArrayView.RandomNormal<ElementType>(new[] { dim, inputDimension }, NormalMean, StandartDev, Seed++, device));
        var bias = new Parameter(NDArrayView.RandomNormal<ElementType>(new[] { dim }, NormalMean, StandartDev, Seed++, device));
        var layer = activationFunction(CNTKLib.Plus(bias, CNTKLib.Times(weights, previousLayer)));
        return layer;
    }
    public static Function LSTM<ElementType>(int cellDim, Variable previousLayer, DeviceDescriptor device)
    {
        Func<Variable, Function> pastValueRecurrenceHook = (x) => CNTKLib.PastValue(x);
        var dh = Variable.PlaceholderVariable(new int[] { cellDim }, previousLayer.DynamicAxes);
        var dc = Variable.PlaceholderVariable(new int[] { cellDim }, previousLayer.DynamicAxes);

        var prevOutput = dh;
        var prevCellState = dc;
        int outputDim = prevOutput.Shape[0];

        bool isFloatType = typeof(ElementType).Equals(typeof(float));
        DataType dataType = isFloatType ? DataType.Float : DataType.Double;

        Func<int, Parameter> createBiasParam;
        if (isFloatType)
            createBiasParam = (dim) => new Parameter(new int[] { dim }, 0.01f, device, "");
        else
            createBiasParam = (dim) => new Parameter(new int[] { dim }, 0.01, device, "");

        uint seed2 = 1;
        Func<int, Parameter> createProjectionParam = (oDim) => new Parameter(new int[] { oDim, NDShape.InferredDimension }, dataType, CNTKLib.GlorotUniformInitializer(1.0, 1, 0, seed2++), device);

        Func<int, Parameter> createDiagWeightParam = (dim) => new Parameter(new int[] { dim }, dataType, CNTKLib.GlorotUniformInitializer(1.0, 1, 0, seed2++), device);

        Func<Variable> projectInput = () => createBiasParam(cellDim) + (createProjectionParam(cellDim) * previousLayer);

        // Input gate
        Function it =
            CNTKLib.Sigmoid((Variable)(projectInput() + (createProjectionParam(cellDim) * prevOutput)) + CNTKLib.ElementTimes(createDiagWeightParam(cellDim), prevCellState));
        Function bit = CNTKLib.ElementTimes(it, CNTKLib.Tanh(projectInput() + (createProjectionParam(cellDim) * prevOutput)));

        // Forget-me-not gate
        Function ft = CNTKLib.Sigmoid((Variable)(projectInput() + (createProjectionParam(cellDim) * prevOutput)) + CNTKLib.ElementTimes(createDiagWeightParam(cellDim), prevCellState));
        Function bft = CNTKLib.ElementTimes(ft, prevCellState);

        Function ct = (Variable)bft + bit;

        // Output gate
        Function ot = CNTKLib.Sigmoid((Variable)(projectInput() + (createProjectionParam(cellDim) * prevOutput)) + CNTKLib.ElementTimes(createDiagWeightParam(cellDim), ct));
        Function ht = CNTKLib.ElementTimes(ot, CNTKLib.Tanh(ct));

        Function c = ct;
        Function h = (outputDim != cellDim) ? (createProjectionParam(outputDim) * ht) : ht;

        var LSTMCell = new Tuple<Function, Function>(h, c);

        var actualDh = pastValueRecurrenceHook(LSTMCell.Item1);
        var actualDc = pastValueRecurrenceHook(LSTMCell.Item2);

        (LSTMCell.Item1).ReplacePlaceholders(new Dictionary<Variable, Variable> { { dh, actualDh }, { dc, actualDc } });

        Function LSTMFunction = LSTMCell.Item1;

        Function layer = CNTKLib.SequenceLast(LSTMFunction);
        return layer;
    }
}`

Answer: I found this example which helped me a lot: https://github.com/albertalrisa/cntk-csharp-rnn/blob/master/playground/CNTKRefresh/CharRNNCNTK/Program.cs
The input sequence is stored as a sequence of input vectors, for example, to store an input sequence of 10 input vectors of length 5, you need to combine all the input vectors into one vector of length 50 elements.
Value.CreateSequence is used to create a single input sequence, and Value.CreateBatchOfSequences is used to create inputs that include multiple sequences.
I thought that Value.Create only accepts IEnumerable<NDArrayView>, and NDArrayView in the constructor accepts a one-dimensional float[] array, that is, it will not be possible to pass several sequences into it. To do this, you need to use List<List<float>> and pass it to Value.CreateBatchOfSequences.
Also, the sequence length axis is a dynamic axis because it can have a variable length. In the example it is created as:
var axis = new Axis("inputAxis");
var inputVariable = Variable.InputVariable(inputShape, DataType.Float, "inputVariable", new List<Axis> { axis, Axis.DefaultBatchAxis() });
var outputVariable = Variable.InputVariable(outputShape, DataType.Float, "outputVariable", new List<Axis> { axis, Axis.DefaultBatchAxis() });

But it also works without it, I don't know why.
As a result, creating several input sequences with random values and running them through a recurrent neural network will look like this:
var cpu_device = DeviceDescriptor.CPUDevice;
var gpu_device = DeviceDescriptor.GPUDevice(0);
var model_device = gpu_device;

int inputDim = 5;
int cellDim = 5;
int outputDim = 3;
int sequenceLength = 50;
int sequencesCount = 100;

NDShape inputShape = NDShape.CreateNDShape(new int[] { inputDim });
NDShape outputShape = NDShape.CreateNDShape(new int[] { outputDim });

var axis = new Axis("inputAxis");
var inputVariable = Variable.InputVariable(inputShape, DataType.Float, "inputVariable", new List<Axis> { axis, Axis.DefaultBatchAxis() });
var outputVariable = Variable.InputVariable(outputShape, DataType.Float, "outputVariable", new List<Axis> { axis, Axis.DefaultBatchAxis() });

var lstmLayer = CntkWrapper.Layers.LSTM<float>(cellDim, inputVariable, model_device);
var model = CntkWrapper.Layers.Dense<float>(outputDim, lstmLayer, CNTKLib.Sigmoid, model_device);

Random random = new Random();
List<List<float>> inputSequences = new List<List<float>>();
for (int i = 0; i < sequencesCount; i++)
{
    List<float> sequence = new List<float>();
    for (int k = 0; k < sequenceLength; k++)
    {
        float[] input_vector = new float[inputDim];
        for (int n = 0; n < input_vector.Length; n++)
        {
            input_vector[n] = (float)random.NextDouble();
        }
        sequence.AddRange(input_vector);
    }
    inputSequences.Add(sequence);
}

var gpuInputSequences = Value.CreateBatchOfSequences(inputShape, inputSequences, gpu_device);

var inputDataMap = new Dictionary<Variable, Value>() { { inputVariable, gpuInputSequences } };
var outputDataMap = new Dictionary<Variable, Value>() { { model.Output, null } };
model.Evaluate(inputDataMap, outputDataMap, gpu_device);
var outputValue = outputDataMap[model.Output];
IList<IList<float>> actualLabelSoftMax = outputValue.GetDenseData<float>(model.Output);


Extensions of fields and dimension

Question: The following is a homework question:

Let M: N be a field extension, with a ∈ M algebraic over N. Show every element of N(a) is algebraic over N.

Can anyone give me a strategy to approach this?
Answer: If an element $\alpha$ is algebraic over a field $F$, then it is a root of an irreducible polynomial $f(x) \in F[x]$.  Let's say that polynomial has degree $n$.
It follows that $F[\alpha]$ is a degree $n$ field extension over $F$.  Now think about how "degree" is defined in terms of field extensions as vector spaces over the base field: if $[F[\alpha]:F] = n$, then  $\{1, \beta, \beta^2, ..., \beta^{n} \}$ cannot be a linearly independent set for any $\beta \in F[\alpha]$.  This means, for any $\beta \in F[\alpha]$, we can write:
$$\sum_{k=0}^{n} c_k\beta^k = 0\ \ \ \text{ for } \ c_k \in F \text{ not all zero}$$
How can you use this information to construct a polynomial for which $\beta$ is a root?

Extensions of fields and dimension