Forwarding ssh agent to container with Podman in Fedora

Question: I am creating a podman image and I'm trying to understand how to allow the container use my host (Fedora 37) ssh-agent so as the container can clone git repositories. I would prefer this than using PAT or other solutions that I can think of.
I have set-up a very basic example:
The application foo:
#!/bin/bash -eux
git clone "$REPO"

Dockerfile:
FROM docker.io/fedora:39@sha256:531209ee9007c2bf909c6548b0456524bde9da0a1ac95ecc67bf3b98e9046969
ARG HOST
RUN dnf install -yq git
WORKDIR /app
COPY foo foo
ENV PATH="${PATH}:/app"
RUN useradd -m user && mkdir -p /home/user/.ssh && chown -R user:user /home/user/.ssh
WORKDIR /home/user
RUN printf "Host $HOST\n\tStrictHostKeyChecking no\n" >> .ssh/config
USER user
ENTRYPOINT ["foo"]

Then I build the image:
podman build -t foo --build-arg HOST=$HOST .

And try to run it with:
$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -eREPO=$REPO foo

 + git clone git@*********.git

...
*****: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

If I run the container with --entrypoint /bin/bash and execute ssh-add -L I get:
[user@42e1646bd254 ~]$ ssh-add -L
Could not open a connection to your authentication agent.

I noticed that the "$SSH_AUTH_SOCK" dir in the container is owned by root:
[user@246199390bfe ~]$ stat $SSH_AUTH_SOCK
  File: /run/user/1000/keyring/ssh
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 0,68    Inode: 81          Links: 1
Access: (0755/srwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-19 07:28:45.436018916 +0000
Modify: 2023-03-18 06:20:31.860105581 +0000
Change: 2023-03-19 10:17:03.933082078 +0000
 Birth: 2023-03-18 06:20:31.860105581 +0000

And reading documentation I understood that podman runs as root within the container but gets mapped to my user outside the container (or something like that). So I tried to change my Dockerfile to use root instead of user:
FROM docker.io/fedora:39@sha256:531209ee9007c2bf909c6548b0456524bde9da0a1ac95ecc67bf3b98e9046969
ARG HOST
RUN dnf install -yq git
WORKDIR /app
COPY foo foo
ENV PATH="${PATH}:/app"
RUN mkdir -p /root/.ssh
WORKDIR /root
RUN printf "Host $HOST\n\tStrictHostKeyChecking no\n" >> .ssh/config
ENTRYPOINT ["foo"]

Re-built and then tried something like:
podman run -it -v $SSH_AUTH_SOCK:/root/keyring:Z -e SSH_AUTH_SOCK=/root/keyring -eREPO=$REPO foo

But I still get a Permission denied (publickey).
I've checked that stat $SSH_AUTH_SOCK in the host contains Context: system_u:object_r:container_file_t:s0:c235,c239 (so I believe the :Z option is working as intended.
Can anyone help me understand what I'm doing wrong?
Using Podman 4.4.2 on Fedora.
UPDATE
I uninstalled podman and installed docker-ce and it worked on first attempt (tried the modified Dockerfile).
I would prefer to stay on podman though.
Comment: Try mapping the dir not the file. `-v "$(dirname "$SSH_AUTH_SOCK")":/root/keyring:Z -e SSH_AUTH_SOCK=/root/keyring/ssh`
Answer: I struggled with this problem too but I found a solution.
The key is adding --privileged in your run (let's leave security concerns for now)
Here is my Dockerfile:
FROM fedora:37

RUN dnf install -yq git && dnf clean all
RUN mkdir -p .ssh
RUN printf "Host *\n\tStrictHostKeyChecking no\n" >> .ssh/config
ENTRYPOINT ["/bin/bash"]

Build with the following:
podman build -t foo:temp

And ran with the following run:
$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK --privileged foo:temp -c "ssh -T firstname.lastname@example.com"
The authenticity of host 'github.com (126.96.36.199)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi op-jenkins! You've successfully authenticated, but GitHub does not provide shell access.

Idk why StrictHostKeyChecking no does not work, but once you just say yes, it can do ssh transaction.
I also tried with --cap-add ALL but that somehow failed:
$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK --cap-add ALL foo:temp -c "ssh -T firstname.lastname@example.com"
The authenticity of host 'github.com (18.104.22.168)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
firstname.lastname@example.com: Permission denied (publickey).

When I turned -v option in ssh -vT email@example.com, along the debug messages, I did find this error:
debug1: pubkey_prepare: ssh_get_authentication_socket: Permission denied

So adding all linux privileges via cap-add is not enough somehow to get permissions.
I feel that there is some security lift up when doing --privileged but I am no expert. And in my host machine, we have SELinux enabled... so that might also play a factor...? idk.
But hopefully this helped.


SQL Query to find rows based on descriptions

Question: I am running this query in MySQL:
SELECT sequence
from prices
WHERE match_description <> ''
  AND 'SIP Trunk: 123456 (1 Channel)' LIKE CONCAT(match_description, '%')
ORDER BY length(match_description) desc
LIMIT 1;

I have rows that have a match_description of:
SIP Trunk (1 Channel)
SIP Trunk (2 Channels)
SIP Trunk (3 Channels)

etc...
when running the above query, the string looks like SIP Trunk: 123456 (x Channels)
how can i match these with my rows correctly?
bearing in mind, the query i have works with other strings so i cannot change it too much.
for example, other queries that run are like:
SELECT sequence from prices WHERE match_description <> '' AND 'Seat 200' LIKE CONCAT(match_description, '%') ORDER BY length(match_description) desc LIMIT 1;

this will match a row with match_description of Seat
the queries are run when i upload a CSV file. this includes product names that are supplied to me so they are uploaded as they are sent to me.
some example rows would be:
Seat 200
Seat 201
Seat 202
Call Queue 200
Call Group 201
Geographic Number 01234 567890
SIP Trunk: 123456 (2 Channels)
SIP Trunk: 654321 (5 Channels)

So, using my query, I am able to match all except for the last two.
for the ones I can match, i have the following in match_description in my table:
Seat
Seat
Seat
Call Queue
Call Group
Geographic Number

i am just unsure how to match these? (the 123456 and 654321 are different for each product)
SIP Trunk: 123456 (2 Channels)
SIP Trunk: 654321 (5 Channels)

Comment: I don't understand your question. It sounds like you want the functionality of `like` to change, based on the content of the pattern.  That is not how SQL works.  You need to bring in the pattern that you want to match.
Comment: can you post the incorrect rows matched?
Comment: Do you need to match just SIP Trunk:  (x Channel) or SIP Trunk:  (x Channel) first and then then number 123456. You can just ignore 123456 part by doing some string operations in query or in your csv if you don't need it.
Comment: the uploaded file will be `SIP Trunk: 123456 (x Channels)` but in my database, i will not have these different numbers stored. all i have is a products table that has a product for each number of channels -  `SIP Trunk: (x Channels)`
Answer: You could match the first 11 characters to SIP Trunk:, then the last 9/ 10 characters to channel) or channels) then that way, if your match_description is of the form: SIP Trunk: yyyyyy (x Channels) it will pick it up. The MID statements will make sure that x and yyyyyy are integers and the opening bracket is there.
Also, thanks to @dwjv who points out you won't be able to index on the match_description column.
SELECT
    sequence
FROM
    prices
WHERE
    match_description <> ''
AND
    LEFT(match_description, 11) = 'SIP Trunk: '
AND
    (RIGHT(match_description, 9) = ' channel)' OR
     RIGHT(match_description, 10) = ' channels)')
AND
    MID(match_description, 12,6) REGEXP '^-?[0-9]+$'
AND
    MID(match_description, 18,3) = ' ( '
AND
    MID(match_description, 20,1) REGEXP '^-?[0-9]+$'
ORDER BY
    length(match_description) desc
LIMIT 1;

Comment: probably enough, but you could also add in `AND MID(match_description, 12,1) IN (1,2,3,4,5,6)`
Comment: Good shout, put in a little `REGEXP` to check it parses as an integer.
Comment: Nice :-). Probably worth pointing out this won't be able to use any indexes on the match_description column though.
Answer: Try something like this
SELECT sequence
from prices
WHERE match_description <> ''
  AND match_description LIKE  '%SIP Trunk: 123456 (% Channel%)%'
ORDER BY length(match_description) desc
LIMIT 1;

Answer: You can change the string to match to your match_description
For example you can encode SIP Trunk 12345(x Channel) to SIP Trunk (x Channel) 12345
This way your values will match with string.
SIP Trunk (1 Channel)
SIP Trunk (2 Channels)
SIP Trunk (3 Channels)

For encoding purpose you may use case statement on the variable that gives you specific type of string.
Do you need to match just SIP Trunk: (x Channel) or SIP Trunk: (x Channel) first and then number 123456. You can just ignore 123456 part by doing some string operations in query or in your csv if you don't need it.


In Firebase, how do I check if a user has write access to the database? or how do I restrict a user from accessing the database?

Question: I want something like this:
firebase.auth().currentUser.canWrite();

This should return true or false based on the security rules I have made in Firebase Console.
Answer: First you need to restrict access to your realtime database using custom claims and security rules (since by default, you restrict or allow read/write permissions for all users).
Then, on the client, check that a given user has access to the db by parsing the id token and checking the custom claims (refer to example in section "Access custom claims on the client" from the link above. Example is written in JS, but there should be no problem converting it to e.g. Swift). Further, you could easily wrap this functionality (check for valid claims) in your own canWrite() extension on the FIRUser object which returns true or false depending on if token has the correct claims.
Comment: Cool. I had a look, and I've managed to find a workaround that suits my purpose. I've posted it as an answer [link] (https://stackoverflow.com/a/47210909/8905779)here. Thanks!
Answer: Okay, I went through the suggestion, and after analyzing everything I've found a sweet workaround. Now let me first tell you that I wanted to restrict users other than the admin/s from seeing UI buttons like 'Add new Item', 'Delete item' etc., so that they cannot modify the database.
for example, I want only the user with credentials email@example.com to have access to buttons and pages that allow modification in database.
Ps. This is an Ionic project.
In my app.component.ts file:
import { Events } from 'ionic-angular';

export class MyApp{

  admins: any;

  constructor(public events: Events){

    this.initializeApp();

    this.admins =[
      {email: 'email@example.com'},
      {email: 'email@example.com'}
    //You may add more users whom you want to give admin privilege..
    ];

   // Now subscribe to an event to check if the logged-in user is an admin user or not!

    events.subscribe('check:admin', (email) => {
      var isAdmin = false;
      this.admins.forEach((x) => {
        if(email === x.email)
          isAdmin = true;
      });
     return isAdmin;
    });
  }
}

In my home.ts file: This can be any page in your project, not necessarily homepage.
import { Events } from 'ionic-angular';

export class HomePage{

  // Now declare and initialize a boolean variable. This will be true if the logged-in (currentUser) user is an admin.

    isAdmin = this.events.publish('check:admin', firebase.auth().currentUser.email)[0];
 // this.events.publish() returns an array, hence the weird workaround of '[0]'.

  constructor(private events: Events){}

}

Now in my home.html page:
<button *ngIf="isAdmin">Add new Item</button>

Here, the 'Add new Item' button will only show if the variable 'isAdmin' is true. This way, we can have control over which UI elements to display to which type of users..
Hope this helps. Thanks!
Answer: A possible way to check whether user has access to the database is to try to read (or write) any value from the database.
For example, I have a "dummy" node, and I try to read the value. If the operation is not successful, this means that I'm dealing with a permission issue:
dummyNode.addListenerForSingleValueEvent(new ValueEventListener() {
            @Override
            public void onDataChange(DataSnapshot dataSnapshot) {
                // read access successfully tested.

            }

            @Override
            public void onCancelled(DatabaseError firebaseError) {

                Log.v(TAG,"firebaseError:"+firebaseError);

                //Common Error here: Transaction at /F failed: DatabaseError: Permission denied

            }
        });


calculation of $E[\Phi(X)]$

Question: Let, $X\sim N(\mu, \sigma^2)$ and let $\Phi(\cdot):\mathbb{R}\to[0,1]$ be the CDF of a standard normal distribution.
Then, what is the pdf of $Y=\Phi(X)$. Also, find $E[\Phi(X)]$.
Note:-
Here, $Y=\Phi(X)\ne P(Z\le X)= \text{ some constant }$. Rather, in this case, $Y=\Phi(X)$ is a non-degenerate random variable.
And, so $E[\Phi(X)]\ne E[P(Z\le X)]=P(Z\le X)=\Phi\left(\dfrac{-\mu}{\sqrt{1+\sigma^2}}\right)$
My attempt:-
$$E[Y]=E[\Phi(X)]=\int^{\infty}_{-\infty}\Phi(x)f_X(x)dx=\int^{\infty}_{-\infty}\left( \int^{x}_{-\infty} \dfrac{1}{\sqrt{2\pi}}\exp\left( \frac{-1}{2}t^2 \right)dt \right) \dfrac{1}{\sigma\sqrt{2\pi}}\exp\left( \frac{-1}{2}{\bigg[\dfrac{x-u}{\sigma}}\bigg]^2 \right)dx \\\ =\dfrac{1}{2\pi\sigma}\int^{\infty}_{-\infty}\int^{x}_{-\infty}\exp\left[\dfrac{-1}{2}\Bigg(t^2+\Big(\frac{x-\mu}{\sigma}\Big)^2\Bigg)\right]dt dx $$
This is where i am stuck. Any help would be appreciated.
Comment: It is true that $$\mathbf{E}[\Phi(X)]=\mathbf{P}(Z\leq X)$$ where $Z$ is a standard normal random variable independent of $X$. This is because we can write $\Phi(X)=\mathbf{P}(Z\leq X\mid X)$.
Comment: $$\underset{X}{\mathop{\mathbb{E}}}\,\left[ \Pr \left[ Z<X\left| X \right. \right] \right]=\underset{X}{\mathop{\mathbb{E}}}\,\left[ \underset{Z}{\mathop{\mathbb{E}}}\,\left[ {{I}_{\left\{ Z<X \right\}}}\left| Z \right. \right] \right]=\mathbb{E}\left[ {{I}_{\left\{ Z<X \right\}}} \right]=\Pr \left[ Z<X \right].$$   Taken from https://math.stackexchange.com/a/4140516/415432 and https://math.stackexchange.com/a/1125935/415432
Answer: The CDF of $X$ is
$$
\mathbb P(X\le x)=\mathbb P\Bigg(\frac{X-\mu}{\sigma}\le \frac{x-\mu}{\sigma}\Bigg)=\Phi\Big(\frac{x-\mu}{\sigma}\Big).
$$
Therefore
$$
U:=\Phi\Big(\frac{X-\mu}{\sigma}\Big)
$$
is uniform on $[0,1]$. From $Y=\Phi(X)$ we get
$$
Y=\Phi\Big(\sigma\,\Phi^{-1}(U)+\mu\Big)\,,\quad U=\Phi\Bigg(\frac{\Phi^{-1}(Y)-\mu}{\sigma}\Bigg)\,.
$$
The CDF of $Y$ is
$$
\mathbb P(Y\le y)=\mathbb P\Bigg(U\le\Phi\Bigg(\frac{\Phi^{-1}(y)-\mu}{\sigma}\Bigg)\Bigg)=\Phi\Bigg(\frac{\Phi^{-1}(y)-\mu}{\sigma}\Bigg)\,.
$$
The expectation of $Y=\Phi(X)$ is
$$
\mathbb E[Y]=\int_0^1\Phi\Big(\sigma\,\Phi^{-1}(u)+\mu\Big)\,du\,.
$$

calculation of $E[\Phi(X)]$


Is every puck used in a game given to a player?

Question: I'm used to see the puck used to score a goal be given to the player that scored the goal; but this seems to be something else. https://twitter.com/TomasPleky14/status/1103306786965278721
Carey Price got what I would guess is the puck that was used in the last seconds of play for his 314th victory.
So it sparks the question: Is every single puck used in a game given to a player playing the game? Or are those photos just for big events (Tie of a record, hat tricks, first goal, etc)
Answer: The pucks are often only for show and sometimes sold or auctioned later. Some players keep them, some don't.
During a game about 10-15 pucks (or even more) are used and not all of them are given away to players. Some will be kept or given to fans and unusable/damaged pucks get discarded.


Unimaginable speed of sockets into Oracle JVM

Question: I'm working with Oracle and I'm asked to speed up some legacy application. For this speedup I'm in need of some in-memory singleton containing temporary data; currently these data are stored into database permanent tables and it imposes much load to database. Temporary tables cannot be used due to multi-session access to data.
I tried to make this singleton using stored Java. Main problem I found - Oracle makes something like separate JVM for every database session (https://docs.oracle.com/cd/B28359_01/java.111/b31225/chtwo.htm#BABBDCDI), so singleton cannot be accessed via common memory.
I makes some API containting "server" (singleton) and "client" (access methods). Client communicates with server via local sockets. Because single-threaded nature of stored Java (https://docs.oracle.com/cd/B28359_01/java.111/b31225/chtwo.htm#BABHHHDG) server accepts and processes requests into a single thread like this (simplified):
<code>    ServerSocket socket = new ServerSocket(..., InetAddress.getByName(null));
    while (!finished) {
      Socket client = socket.accept();
      ..
      output.println(process(input));
      ..
      client.close();
    }
</code>
Client sends request like this:
<code>  Socket socket = new Socket(InetAddress.getByName(null), getPort(requestId));
  ..
  for (int i = 0; i < data.length; i++) output.println(data[i]);
  String result = in.readLine();
  ..
  socket.close();
</code>
While runnings tests as a stand-alone Java application, without Oracle, it works like this on my workstation:

34001 calls served at 3.178 sec.
Avg. 10698.86721208307 calls per second
Avg. wait time: 0.03199905885121026 ms.
Avg. process time: 0.0628510926149231 ms.

It's generally acceptable. But when loading into Oracle, such test shows a very different speed:

117 calls served at 24.218 sec.
Avg. 4.831117350730861 calls per second
Avg. wait time: 90.17094017094017 ms.
Avg. process time: 203.54700854700855 ms.

Why it's so slow? What can I do to repair this nonsense? Is there better way to communicate between sessions in Oracle/jvm?
Comment: Slowness is likely a "you-problem" or more specifically a problem of your specific environment and as such it is hard to say what to do when there is nothing to actually go on. No configuration details, no environment description, no nothing. For all we know you're running the database in a virtual machine with the I/O throughput severely limited. You also don't detail what you are communicating between the databases, so it is also impossible to give any advice about what might be a better strategy (plus that is opinionated). Oracle Advanced Queuing comes to mind, as a wild guess.
Comment: The OS the oracle DB runs under also has a lot of bearing on performance.  Your data as presented does not mean much in terms of getting help or suggestions.  Do you have perf stats for the OS - I/O queue request lengths, etc.?   This clearly looks like a possible OS or listener issue.  Check the listener log on the oracle machine.
Comment: Does your latest attempt still use Java on the database server? What is the value of the java_pool_size parameter?
Comment: Thank you. I guessed it's my problem but now it really looks like environment problem. Measured in several environments, it shows speed from 15'000 standalone / 5'000 Oracle (my home computer) via 250-500 Oracle (some our servers) to 4-5 Oracle (extreme case on two our servers). So our DBAs will research cause of it.
Answer: As a result of investigation.

Administrators said nothing interesting about i/o quests requests length etc.
Oracle's internal java is several times slower than stand-alone one. I wrote a simple example, in which converts integer to string and put it into map; it works about to 4 time slower in stored procedure. Penalty is different for various operations.
I profiled class. Using different techniques (nodelay, reusing sockets, threading) I reached about to 8 times faster in stand-alone application and about to 100 times faster in stored Java, it was limit I can do.
At this moment I refused of stored-java-only. I started an stand-alone application with server part near the database and set up client part into database to connect it. It grants me speed I required; in such case effective speed is about 80% of stand-alone application. This speed is enough for me; it stopped to be a bottleneck.


Call method in Parallel.Foreach and save the result into the thread safe list

Question: I have this code which is running fine in single thread
foreach (var meetingRoom in meetingRooms)
{
            try
            {
                var res = calendarService.GetEvents(meetingRoom.Email, startUtc, endUtc, lightweight, limit);
                result.AddRange(res);
            }
            catch (Exception ex)
            {
                Log.Error(ex);
            }
}

but the issue is the meetingRooms list contains more than 40 000 entries. I am trying to approach the Parallel.Foreach - this is what I´ve got so far:
Parallel.ForEach(meetingRooms, () => 0,(meetingRoom) =>
        {
            var res = calendarService.GetEvents(meetingRoom.Email, startUtc, endUtc, lightweight, limit);

        }
        ,
        finalResult =>
        {
            lock (result)
            {
                result.Add(finalResult);
            }
        });

I do not have luck with the understanding of how the fourth parameter localFinally works.
Please note that result and meetingRooms are the lists with different datatypes.
EDIT:
I´ve implemented the PLINQ solution as one of answers indicates. Seems like its working but the Exchange Web Services(EWS) library is not implemented for parallel calling of one of its method. Getting and error "Not a unique key!" which is thrown by Microsoft code.
Comment: There isnt really a question here - I dont see a pramater of "localFinally" .. what specifically is the issue
Comment: What context is this in? ASP.NET, console, WPF, windows forms?
Comment: The samples are onconsistent in the use of `result.Add()` and `result.AddRange()`, better fix that.
Answer: You can use PLinq.
result = meetingRooms
  .AsParallel()
  .Select(meetingRoom => calendarService.GetEvents(meetingRoom.Email, startUtc, endUtc, lightweight, limit) )
  .ToList();

When you still want to use Parallel.ForEach(), the overloads with localFinally aren't so useful. Use the basics:
Parallel.ForEach(meetingRooms, meetingRoom =>
    {
        var res = calendarService.GetEvents(meetingRoom.Email, ...);

        lock (result) // only OK when you don't lock on it anywhere else
        {
            result.Add(res);
        }
    });

Comment: I don't understand why you think that the ForEach-version with local state isn't useful. Isn't it used so that each partition can have its local state which can be accessed without locking?
Comment: Given that there are 40000 meeting rooms, your version would invoke the lock statement for each iteration of the loop. A version with local state would lock only once per partition, which is considerably less. It might not matter much, if the call to GetEvents takes far more time than acquiring and releasing the lock, but that's up to the OP to discover.
Comment: Why do you think that would be useful? It can be used when your intermediate results are very small.  When GetEvents() does something substantial the localResults won't make much difference,
Comment: An uncontested lock is a matter of nanoseconds, this has been discussed here before. Complicating your source to solve a non-existing problem is always much worse.
Answer: You may also use a ConcurrentBag (or ConcurrentQueue or ConcurrentDictionary) to store the results and avoid the lock.
Comment: *Not* a concurrent bag, that's for fast thread-*local* storage.
Comment: Depends on the needs of the application. In this case adding items is faster with a ConcurrentBag, and reading them (from the same thread, of course) will use an internal lock.
Comment: @PanagiotisKanavos - "concurrent bag is for thread-local storage" ? Huh? That needs a citation.
Comment: @HenkHolterman docs, source code or Stephen Toub blog post? The data added by each thread is stored in TLS for faster access by its own thread. Reading other threads' data results in cross-thread operations
Comment: @HenkHolterman for example [FAQ :: Are all of the new concurrent collections lock-free?](https://blogs.msdn.microsoft.com/pfxteam/2010/01/26/faq-are-all-of-the-new-concurrent-collections-lock-free/) from 2010 `it maintains a local queue for each thread that accesses it, and under some conditions, a thread is able to access its local queue in a lock-free manner with little or no contention.  Therefore, while ConcurrentBag sometimes requires locking, it is a very efficient collection for certain concurrent scenarios (e.g. many threads both producing and consuming at the same rate). `
Comment: Pick one. I know what TLS is, I'm curious about what role you see for a ConcurrentBag.
Comment: How about all? FAQ was first. Stephen Toub links to [Performance of Concurrent Collections in .NET 4](https://blogs.msdn.microsoft.com/pfxteam/2010/04/26/performance-of-concurrent-collections-in-net-4/) second. Perhaps you should ask the PFX team intended for ConcurrentBag. I believe that article covers it
Comment: That just makes the Bag very well suited for this localResults scenario. So why did your first comment start with "Not..." ? But I just don't buy the need to avoid the lock here at all.
Comment: Why? the threads don't *consume* their own data. The consumer is the thread that ends up *reading* that data. From the perf doc : `it will not have the same behavior for pure producer-consumer scenarios as all the consumers will have to repeatedly perform stealing operations and this will incur significant overheads and synchronization costs.` Page 17. Also note the memory impact
Comment: Henk Holterman, the Concurrent collection are optimized for concurrent access, therefore usually they are faster then locks, plus you don't have to worry about handling multiple locks on the same resource.
Comment: @drevis the problem is that ConcurrentBag is a special-purpose collection. It has different performance characteristics than ConcurrentQueue, ConcurrentStack or ConcurrentDictionary. It's actually more expensive to read from it after the parallel loop finishes
Comment: @PanagiotisKanavos true, that's why I said that depends on the application. You can't exclude one a priori, maybe here the read is not critical.


Add field to an entity_view for dispaly in a tapir table in ubercart

Question: In the simplest terms I am trying to add a 'vendor' column to the order view table in Ubercart which will display the author of each line item product.
Below is the table of interest.  I have hacked the direct code contained in uc_order.order_pane.inc but will refactor with hooks the drupal way ASAP. I have added my comments in the code below.
function uc_op_products_view_table($order) {
  dpm($order);
  $table = array(
    '#type' => 'tapir_table',
    '#attributes' => array('class' => array('order-pane-table')),
  );

  $table['#columns']['qty'] = array(
    'cell' => array(
      'data' => theme('uc_qty_label'),
      'class' => array('qty'),
    ),
    'weight' => 0,
  );
  $table['#columns']['product'] = array(
    'cell' => array(
      'data' => t('Product'),
      'class' => array('product'),
    ),
    'weight' => 1,
  );

Added by me:
  $table['#columns']['Vendor'] = array(
    'cell' => array(
      'data' => t('Vendor'),
      'class' => array('vendor'),
    ),
    'weight' => 1.5,
  );

.
  $table['#columns']['model'] = array(
    'cell' => array(
      'data' => t('SKU'),
      'class' => array('sku'),
    ),
    'weight' => 2,
  );
  if (user_access('administer products')) {
    $table['#columns']['cost'] = array(
      'cell' => array(
        'data' => t('Cost'),
        'class' => array('cost'),
      ),
      'weight' => 3,
    );
  }
  $table['#columns']['price'] = array(
    'cell' => array(
      'data' => t('Price'),
      'class' => array('price'),
    ),
    'weight' => 4,
  );
  $table['#columns']['total'] = array(
    'cell' => array(
      'data' => t('Total'),
      'class' => array('total'),
    ),
    'weight' => 5,
  );

  if (!empty($order->products)) {
    $build = entity_view('uc_order_product', $order->products);

This is the entity I want to edit. uc_order_product needs to have a field for Author / Vendor to make my table look right.
    dpm($build);
    $table['#rows'] = $build['uc_order_product'];
  }
  else {
    $table['#rows'][]['product'] = array(
      '#markup' => t('This order contains no products.'),
      '#cell_attributes' => array('colspan' => 'full'),
    );
  }

  return $table;
}

Answer: So this is the ugly hack I used to resolve the issue:
  if (!empty($order->products)) {
    $build = entity_view('uc_order_product', $order->products);
    foreach($build['uc_order_product'] as $line_item) {
      $node = node_load($line_item['#entity']->nid);
      $vendor['#markup'] = 'uid) . '>' . $node->uid . '';
      $vendor['#cell_attributes']['class'][] = 'vendor';
      $line_item_id = $line_item['#entity']->order_product_id;
      $build['uc_order_product'][$line_item_id]['vendor'] = $vendor;
    }
    $table['#rows'] = $build['uc_order_product'];
  }


Why does matching with replacement result in lower efficiency?

Question: In the following paper, https://users.nber.org/~rdehejia/papers/matching.pdf, it says:

Matching with replacement involves a tradeoff between bias and
variance. With replacement, the average quality of matching
increases; thus, the bias decreases but the variance increases.

Why is it that the variance increases? Is there a simple and intuitive way to understand this? I understand the reuse of control units results in less information being used, but is there a mathematical or intuitive way to understand the link from this to the higher variance? Thanks.
Answer: I can offer two answers, one intuitive but somewhat unsatisfying and the other statistically rigorous but (to me) somewhat opaque.
The first comes from a measure known as the effective sample size (ESS) of a weighted sample, which is the approximate size of an unweighted sample that carries the same information as the weighted sample in question. One can estimate the ATT after matching with replacement by assigning weights to each control unit equal to the total number of times that control unit is matched (it's a little more complicated when matching more than one control unit per treated unit). Given these weights, we can compute the ESS of the control sample as $$
ESS = \frac{(\sum w)^2}{\sum w^2} = \frac{n}{1 + \text{Var}(w^*)}$$
where $w_i^* = \frac{w_i}{\bar{w}}$. The greater the variability of the weights, the lower the ESS is, so the lower the precision of the estimate of the control potential outcomes mean is.
The more technical answer comes from Abadie & Imbens (2006, p246-247), who derive the asymptotic variance of matching estimators. A term in the numerator of the variance of the ATT (and there is a similar term in the variance of the ATE) is
$$\sum\limits_{i=1}^N \left(W_i-(1-W_i)\frac{K_M(i)}{M} \right)^2 \sigma^2(X_i,W_i)$$
where $W_i=1$ if the unit is treated and $0$ otherwise and $K_M(i)$ is the number of times unit $i$ is used as a match, $M$ is the number of matches per treated unit, and $\sigma^2(X_i, W_i)$ is the variance of the potential outcome residual, which we'll assume is constant here so we can ignore it. Because of the square, if any control unit were to be matched more than once, this component of the variance would be higher than if all control units were matched at most once.
Consider a simple example with two control units and two treated units and 1:1 matching (so $M=1$). If both control units were matched once (i.e., each to one treated unit), the variance component contribution of the control units ($W_i=0$) would be $(-1)^2 +(-1)^2 = 2$. If one were matched zero times and the other were matched twice, the component would be $(0)^2 + (-2)^2 =4$. So, clearly, by reusing control units when matching with replacement, the variance increases relative to matching each unit only once.

Abadie, A., & Imbens, G. W. (2006). Large Sample Properties of Matching Estimators for Average Treatment Effects. Econometrica, 74(1), 235–267. https://doi.org/10.1111/j.1468-0262.2006.00655.x
Answer: I'm not sure I understand details of your 'matching' procedure. However, a binomial distribution (with replacement) has larger variability than a corresponding hypergeometric distribution (without replacement). Intuitively, as sampling without replacement depletes the population, the variability of available choices decreases.
Example: Consider the number of red chips in five draws, sampling with replacement (binomial) from an urn with 5 red chips and 5 green chips.
By contrast, consider sampling without replacement (hypergeometric). The
two distributions are plotted below. The binomial standard deviation is $1.1180$ and the hypergeometric SD is $0.8333.$
x1 = 0:5;  n = 5;  p = 0.5
b.pdf = dbinom(x1, n, p)
x2 = 0:5;  n = 5;  r = 5; b = 5
h.pdf = dhyper(x2, r,b, n)

hdr = "Binomial (blue) and Hypergeometric Distributions"
plot((0:5)-.02, b.pdf, ylim=c(0,.4), ylab="PDF", xlab="x",
     type="h", lwd=2, col="blue", main=hdr)
 lines((0:5)+.02, h.pdf, type="h", lwd=2, col="brown")
 abline(h=0, col="green2")

Below each of the experiments is simulated 10,000 times and results
are summarized.
set.seed(2021)
x1 = rbinom(10^4, 5, .5)
summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  0.000   2.000   3.000   2.515   3.000   5.000
[1] 1.123965   # aprx binomial SD
x2 = rhyper(10^4, 5,5, 5)
summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  0.000   2.000   3.000   2.497   3.000   5.000
[1] 0.8397981


Any docs on what rights need to be given to do a thing on kubernetes?

Question: Here my first ServiceAccount, ClusterRole, And ClusterRoleBinding
---
# Create namespace
apiVersion: v1
kind: Namespace
metadata:
  name: devops-tools
---
# Create Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: devops-tools
  name: bino

---
# Set Secrets for SA
# k8s >= 1.24 need to manualy created
# https://stackoverflow.com/a/72258300
apiVersion: v1
kind: Secret
metadata:
  name: bino-token
  namespace: devops-tools
  annotations:
    kubernetes.io/service-account.name: bino
type: kubernetes.io/service-account-token

---
# Create Cluster Role
# Beware !!! This is Cluster wide FULL RIGHTS
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: devops-tools-role
  namespace: devops-tools
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - networking.k8s.io
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# Bind the SA to Cluster Role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: devops-tools-role-binding
subjects:
- namespace: devops-tools
  kind: ServiceAccount
  name: bino
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: devops-tools-role
---

It work when I use to create NameSpace, Deployment, and Service.
But it fail (complain about 'have no right') when I try to create kind: Ingress.
Then I try to add
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: devops-tools-role-binding-admin
subjects:
- namespace: devops-tools
  kind: ServiceAccount
  name: bino
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

and now 'bino' can do all things.
My question is: Is there any docs on what 'apiGroups' and 'resources' need to be assigned so one service account can do some-things (not all-things)?
Sincerely
-bino-
Comment: have you seen this? https://stackoverflow.com/a/55358685/1376536
Answer: You can run this command to determine the apiGroup of a resource:
kubectl api-resources

You will see something like:
NAME        SHORTNAMES    APIVERSION              NAMESPACED   KIND
ingresses   ing           networking.k8s.io/v1    true         Ingress

So you would need to add this to the rules of your ClusterRole:
- apiGroups:
  - "networking.k8s.io/v1"
  resources:
  - "ingresses"
  verbs:
  - "get"


How to create content type Drupal 7

Question: I need to add a new content type to the node/add page.  How can I go about doing this?
Answer: You should try Google before posting questions like this here.
One of the first few results is the following video:
Drupal 7: Creating Content Types
You might find the following videos useful as well:
Getting Started with Drupal 7 Tutorials

How to create content type Drupal 7


custom rangeValidation

Question: Como fazer um range validator onde o Max seja o ano atual + 1 ?
Estou querendo fazer uma validação em dataannotation em minhas models
quero fazer um range validation de 1950 até ano atual +1
Por padrão não é possível fazer usando o range padrão
 [Range(1950, Convert.ToDouble(DateTime.Now.Year+1), ErrorMessage = "O ano deve estar entre {1} e {2}!" )]

não é possível inserir uma variável no valor.
Então tentei expandir a classe RangeAttribute.
public class StringLengthRangeAttribute : ValidationAttribute
{
                public int Min { get; set; }
                public int Max { get; set; }

                public StringLengthRangeAttribute()
                {
                    this.Min = 0;
                    this.Max = DateTime.Now.Year + 1;
                }

                public override bool IsValid(object value)
                {
                    string strValue = value as string;
                    if (!string.IsNullOrEmpty(strValue))
                    {
                        int len = strValue.Length;
                        return len >= this.Min && len <= this.Max;
                    }
                    return true;

                }
}

Mas no lado cliente ele não fez a validação!.
tentei com o exemplo descrito aqui, e também não funcionou.
https://stackoverflow.com/questions/7843334/data-annotations-how-can-i-replace-range-values-with-web-config-values-in-mvc3?lq=1
Answer: Nessa linha:
int len = strValue.Length;

Você esta pegando a quantidade de caracteres que faz parte dessa string e não seu valor, logo o seu return faz algo como:
return 4 >= 0 && 4 <= 2018;

Já que 2017, por exemplo, possui 4 caracteres.
Tente algo como:
if (value is int)
{
    return value >= this.Min && value <= this.Max;
}

Ficando assim o código final:
public class StringLengthRangeAttribute : ValidationAttribute
{
    public int Min { get; set; }
    public int Max { get; set; }

    public StringLengthRangeAttribute()
    {
        this.Min = 0;
        this.Max = DateTime.Now.Year + 1;
    }

    public override bool IsValid(object value)
    {
        try
        {
            int strValue = Convert.ToInt32(value);
            return strValue >= this.Min && strValue <= this.Max;
        }
        catch (Exception)
        {
            Console.WriteLine("Formato inválido");
        }
    }
}

Comment: editei a sua resposta com o código final, veja se era isso? porém funcionou..valeu
Comment: agora a clausula SE sempre sera verdadeira já que caso eu não consiga fazer a conversão vai estourar uma exceção não tratada, mas resolve o caso sim.

custom rangeValidation


Spring Security - CSRF token on GET request

Question: I configured CSRF protection in my Spring 5 (Boot 2) RESTful web service because I want to prevent CSRF attacks (I am using JWT cookie for authentication and authorization). On each request I am getting a new XSRF cookie. The value of that cookie I am sending back through the X-CSRF-TOKEN header.
I noticed that POST, PUT and DELETE methods require the value of this token (in the X-CSRF-TOKEN header) in order to work properly, but GET method works just fine without X-CSRF-TOKEN header.
Is this behavior intentional because GET method should not change state or I did something wrong?
Answer: CSRF token is intentionally excluded from GET to avoid token leakage to a 3rd party


Convert dates with mysql, Microsoft 1900 Date system

Question: A subsystem (which I don't have control over) exports data to a mysql table and stores dates in number of days from 1900 (http://support2.microsoft.com/kb/180162). I need those dates returned as a unix time stamp.
Example of two dates
SELECT date FROM my_table WHERE id = 1 OR id = 2
41315,3921180556
41321,7267013889

Desired output
SELECT some_magic(date) FROM my_table WHERE id = 1 OR id = 2
1360488279
1361035587

I could not find any date-and-time-functions that does the job. Have I missed something?
I want the output from a SELECT of dates to be delivered as this PHP function I wrote.
function UnMicrosoftify($fDateValue = 0, $iDateBase = 1900) {
    if ($iDateBase == 1900) {
        $iMyDateBase = 25569;

        //Adjust for the fictional 29-Feb-1900 (Day 60)
        if ($fDateValue < 60) {
            --$iMyDateBase;
        }
    } else {
        $iMyDateBase = 24107;
    }

    // Perform conversion
    if ($fDateValue >= 1) {
        $utcDays = $fDateValue - $iMyDateBase;
        $iReturnValue = round($utcDays * 86400);
        if (($iReturnValue <= PHP_INT_MAX) && ($iReturnValue >= -PHP_INT_MAX)) {
            $iReturnValue = (integer) $iReturnValue;
        }
    } else {
        $hours = round($fDateValue * 24);
        $mins = round($fDateValue * 1440) - round($hours * 60);
        $secs = round($fDateValue * 86400) - round($hours * 3600) - round($mins * 60);
        $iReturnValue = (integer) gmmktime($hours, $mins, $secs);
    }
    return $iReturnValue;
}

echo UnMicrosoftify((float) str_replace(',','.', "41315,3921180556")) //displays 1360488279 (ie 2013-02-10 09:24:39)
echo UnMicrosoftify((float) str_replace(',','.', "41321,7267013889")) //displays 1361035587 (ie 2013-02-16 17:26:27)

Is it possible to do this with mysql? How?
Answer: If the date is stored as a varchar you have to cast it to a decimal of suitable precision and then it should work as shown in the first version; if it's already a decimal the second version works.
Try this, I think it is what you want:
SELECT
  date AS ExcelSerialDate,

  ADDDATE(ADDDATE(ADDDATE('1899-12-31 00:00',
                          cast(replace(date,',','.') as  DECIMAL(30,12))),
                INTERVAL -1 DAY), INTERVAL (MOD(date,1) * 86400) SECOND)
  AS ConvDate,

  UNIX_TIMESTAMP(
    ADDDATE(
      ADDDATE(
        ADDDATE('1899-12-31 00:00',
                cast(replace(date,',','.') as  DECIMAL(30,12))),
        INTERVAL -1 DAY),
      INTERVAL (
        MOD(cast(replace(date,',','.') as  DECIMAL(30,12)),1) * 86400
      ) SECOND
    )
  )
  AS UnixTS_From_varchar,

  UNIX_TIMESTAMP(
    ADDDATE(
      ADDDATE(
        ADDDATE('1899-12-31 00:00',date2),
        INTERVAL -1 DAY),
      INTERVAL (
        MOD(date2,1) * 86400
      ) SECOND
    )
  )
  AS UnixTS_from_double

FROM my_table;

Sample SQL Fiddle
With your sample data this gives:
EXCELSERIALDATE     CONVDATE             UNIXTS_FROM_VARCHAR UNIXTS_FROM_DOUBLE
41315,3921180556    2013-02-10 00:00:00  1360488279          1360488279
41321,7267013889    2013-02-17 00:00:00  1361121987          1361121987


ld with -Ttext=0x1000 makes the file very big

Question: I have kernel.c file looks like this
void main() {
    char *video_memory = (char *)0xb8000;
    *video_memory = 'X';
}

I compiled and linked with these 2 commands:
gcc -ffreestanding -c ./src/kernel/kernel.c -o kernel.o -m32 -mtune=i386 -fno-pie
ld -o kernel.bin -Ttext=0x1000 kernel.o --oformat binary -m elf_i386 --strip-all -nmagic

It produced a 128MB kernel.bin
However, when I remove -Ttext=0x1000 in ld command, it produced a 120B kernel.bin.
If I understand correctly, -Ttext=0x1000 will add offset 0x1000 to all memory reference. Why did it make such a big difference?
Comment: If you pick a different output format, like the default ELF, what addresses does it pick for the various sections or segments with `-Ttext=0x1000`?  (Use `readelf -a`).  I'm guessing there are two things a long way from each other, and a flat binary just pads with zeros to fill the gap.  Yeah, I tried it, and there's a read-only non-exec segment at `0x080480d4` (about 128MiB from 0x1000), holding the `.note.gnu.property` section.
Comment: IDK how to get rid of that or I'd post this as an answer, but check the manual; a linker script would work but maybe a way to just filter sections.
Comment: Thanks a lot for your reply. Adding this to linker script would do the job
`/DISCARD/ : {
  *(.note.gnu.property)
}`
Comment: first off do not build to binary just yet, build to elf.  then use readelf or objdump or other to show where the loadable sections are.  If you have one at say 0x00000000 and it has 1 byte and one at 0x1000000 and it has one byte then it is going to pad 0x0FFFFFF bytes and create a 0x10000001 byte file.  But if you have one thing at 0x10000000 and one byte at 0x10001000 then it will make a 0x1001 byte file with whatever padding needed.  the -O binary (objcopy) has to start at the lowest address of the sections and fills in and pads to the highest. to create the ouptput
Answer: The answer is in the comment of @Peter Cordes and @old_timer.
Building it to elf first and using readelf -a, I found that there is a segment at 0x080480d4 (about 128MB from 0x1000) holding .note.gnu.property.
So I add this linker script to discard .note.gnu.property
/DISCARD/ : {
    *(.note.gnu.property)
}


ESXi 5 VM Putty session hangs, vSphere client timing out

Question: First of all I believe this is a ESXi issue but let me know if you have seen this.  It started about a year ago when I noticed occasionally when I putty via SSH to my VM guests, if I do anything that makes it to display a lot of things at once, the session will hang and I have to start a new one quite often only to find the same behaviour.  What I meant by display a lot of things can be any of the following:
1) tail -f filename
2) Paste a long command
3) less filename
If I type in one character at a time this won't happen.  I tried searching online and it always point me to flow control settings and the various suggestions I've tried have never been able to resolve the issue.
Since last week, I've noticed I'm not able to connect to my POP3 server from Outlook (it's timing out from Outlook's perspective).
Today I tried to connect to the ESXi via vSphere client and it gives me a time out also.  Exact behavior and error I saw is similar to the one posted at the following URL but the suggested technique also failed to resolve the issue.
http://davidcocke.blogspot.hk/2012/02/unable-to-login-with-vsphere-client.html
Has anyone experienced this before?  Any suggestions on how to troubleshoot this?
Answer: This is probably an MTU issue; i.e. fragmented packets. Are you on a VPN?
Can you describe what path you're taking to connect to these guests from your computer? What's in between you and the server(s)?
Comment: Thanks for commenting.  I just noticed that this is not limited to the ESX and the guests running on it.  I have a separate physical box that is also giving me the same problem.  The path goes from My PC -> Firewall (XTM21) -> Internet Router -> Internet -> Firewall (XTM505) -> Boxes.  And yes between XTM21 and XTM505 it's a BOVPN.
Comment: It's an MTU issue. I don't know how much control you have over the infrastructure, but if there's a dedicated network person, that's where you should start.
Comment: Sorry I'm the only one.  I tried pinging from my PC to the boxes on the other end using packet sizes 1500 up to 3500 bytes and it's fine.  Is this not an indication that this is not a MTU problem?
Comment: Okay I'm unable to paste the ping results here because I can't enter new lines in comments.  So I have started a new question on the maximum allowable MTU size at http://serverfault.com/questions/553963/maximum-mtu-size.  You can see the ping results there.
Comment: Okay so apparently I have to enable the Don't Fragment option and yes the MTU is found to be 1363 bytes.  Am setting my PC to have MTU = 1200.  Thanks!  You saved me!


Read-only root file system - ext3 error?

Question: I have a problem with ext3 filesystem. It's not writable and I can't remount it.
# echo 1 > /file
-bash: /file: Read-only file system
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
# mount -o remount,rw /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only

Can I fix it without server restart?
Comment: Anything in the logs about LVM problems?
Comment: Can you post output of cat /proc/mounts?
Answer: It's probably dropped into read-only mode at the LVM level (lvs will lack a 'w' in the second caracter of the Attr column), so you might be able to flick it back with an lvchange --ignorelockingfailure -p w VolGroup00/LogVol00.  Alternately, the VG may have dropped write access (typically caused by a hardware failure), in this case indicated by a lack of w in the first character of the Attr column in the output of vgs.  That can't be changed at runtime, by the look of it.
If LVM has dropped write perms on either the LV or VG, it's probably for a very, very good reason, and unless you care not for your data, I'd be looking into the root cause rather than trying to bludgeon LVM into allowing writes again.
Comment: Attr for this volume: -wi-ao, lvchange command returns "Locking type 1 initialisation failed."
Comment: Answer extended.
Answer: If this happened since the machine booted, check dmesg or other system logs for reports of anything terrifying going wrong with the hardware (e.g IO errors reported when accessing the underlying disks) such errors can trigger the kernel to put things into read-only mode until someone (you!) comes along to make a more informed decision.
Answer: There is probably filesystem error. Run fsck after umount. This wont require reboot. Try to fsck without using -a or -p.
Comment: Yes, it will require a reboot: it is his root filesystem :P
Comment: My bad. Should have read question properly. :'(


A question on the minimal tensor norm

Question: Given two C*-algebras $A$ and $B$ and let $A_1$ and $B_1$ be their C*-subalgebras. Can we conclude that $A_1 \otimes_\min B_1$ is a subalgebra of $A \otimes_\min B$?
I think that this is not true, however I don't know how to construct a counter-example.
Answer: It is true for the min-norm, because you can construct the minimal tensor product explicitly by using two faithful representations. Namely, if you fix two faithful representations $\pi:A\to B(H)$ and $\mu:B\to B(K)$, then you have
$$
A\otimes_\min B=\overline{\pi(A)\otimes\mu(B)}\subset B(H\otimes K).
$$
Now you can use the restrictions $\pi|_{A_1}$ and $\mu|_{B_1}$ to construct $A_1\otimes_\min B_1$, which then sits naturally in $A\otimes_\min B$.

A question on the minimal tensor norm


How to use userData in addToolTipRect

Question: I am trying to display a tooltip with the addToolTipRect function but I am not sure how to pass the text that should be displayed. The code doesn't work but I think the code illustrates what I want to do:
var thedata : [String:AnyObject] = ["txt" : "Text to display"];
addToolTipRect(lsTooltipRects[0], owner: self, userData: &thedata)

...
override func view(view: NSView, stringForToolTip tag: NSToolTipTag, point: NSPoint, userData data: UnsafeMutablePointer<Void>) -> String
{
    let pUserData = UnsafeMutablePointer<[String:AnyObject]>(data).memory
    return pUserData["txt"] // how do I get the text ???

    return "hello" // works fine
}

I don't know how to convert the [String:AnyObject] back and forth?
Comment: `view.toolTip` https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ApplicationKit/Classes/NSView_Class/#//apple_ref/occ/instp/NSView/toolTip maybe? I have never used it, but it sounds right according to the documentation. I think you will have to set that value though, not using the way you currently pass the data.
Comment: @Kendel: Thanks but I think this is something else. I don't want a tooltip for the complete view. It should only be displayed for a small part of it which works fine already. The problem is displaying the right text.
Answer: I found the problem. The code is working. The problem was the place where thedata was defined. Because only the pointer is passed the variable will be destroyed by the time you use it. So thedata should be defined globally.
Sorry for the inconvenience.

How to use userData in addToolTipRect


Python 2to3 warning  What does it mean?

Question: I am using 2to3 to convert a script. The only warning I get is:
RefactoringTool: Line 716: You should use 'operator.mul(None)' here.

Line 716 of the original script is:
classes = repeat(None)

I don't get where shall I use operator.mul(None). The reference documentation of repeat() (link to docs) shows that I can pass None without any problem. So, what shall I do?
Answer: 2to3 is just getting confused about which repeat you mean.  It thinks you're using operator.repeat in Python 2:
Help on built-in function repeat in module operator:

repeat(...)
    repeat(a, b) -- Return a * b, where a is a sequence, and b is an integer.

instead of itertools.repeat.  That's not a great guess on its part, to be honest, because operator.repeat takes 2 arguments, but that's what it's guessing.  You can see the transformation listed in the docs.
You can avoid the warning by using the fully-qualified itertools.repeat or just ignore it.
Comment: Yes! indeed I am using itertools.repeat(). Now it's clear thank you!

Python 2to3 warning  What does it mean?


Show $(29,x^2+1)\subset\mathbb{Z}[x]$ is not a maximal ideal

Question: I know the maximal ideals of $\mathbb{Z}[x]$ are of the form $(p,f(x))$ where $f(x)$ is irreducible mod $p$. Here, $17$ is a root of $x^2+1$ mod $29$, so $f(x)$ is not irreducible mod $29$. Thus $(29,x^2+1)$ cannot be a maximal ideal. However, I was hoping to find some "more direct" proofs of this, without leveraging as much machinery. Perhaps we can find some ideal properly contained in $\mathbb{Z}$ that also properly contains $(29,x^2+1)$? I was thinking something like $(29,x^2+1,x)$, but haven't been able to show it properly contains $(29,x^2+1)$. Maybe another way to do this is to assume $(29,x^2+1)$ is indeed maximal, and then derive a contradiction from the fact that $\mathbb{Z}[x]/(29,x^2+1)$ is a field? I'd just like to see a way to do this without using the full classification of the maximal ideals of $\mathbb{Z}[x]$.
Comment: What machinery? $\mathbb{Z}[x]/(29,x^2+1)\simeq\mathbb{Z}_{29}[x]/(x^2+1)$ and since $x^2+1$ has a root in $\mathbb Z_{29}$ the quotient ring $\mathbb{Z}_{29}[x]/(x^2+1)$ has zero-divisors, hence it is not a field.
Answer: We have $x^{2}+1 = (x-17)(x+17) + 290$, hence $x^{2}+1 \in (29,x-17)$; hence $(29,x^{2}+1) \subseteq (29,x-17)$. To show that this inclusion is proper, we show that $x-17 \not\in (29,x^{2}+1)$. For the sake of contradiction, suppose $x-17 = 29 \cdot f + (x^{2}+1) \cdot g$ for some $f,g \in \mathbb{Z}[x]$; plug in $x = -17$ to get $-34 = 29 \cdot f(-17) + 290 \cdot g(-17)$ which means $29$ divides $34$, which is a contradiction.


Para que servem os valores padrões do CSS?

Question: Quase todo comando CSS tem seu valor padrão, que já esta sendo empregado por padrão, mesmo que você não o use, por exemplo.
justify-content: flex-start;

Se você não escrever nada por padrão o elemento já vai ficar como flex-start então afinal para que servem os valores padrões, se eles já estão sendo empregados por padrão?
Answer: O padrão existe e está disponível por dois motivos principais:

Caso você aplique um novo estilo e precise voltar para o padrão para exceções a este estilo, ex:

div {
  display: flex;
  justify-content: center;
}
div.start {
  justify-content: flex-start;
}

  Primeiro
  Segundo
  Terceiro

  Primeiro
  Segundo
  Terceiro

Quem aplica o padrão é o navegador, portanto se você não fizer uma definição terá que confiar que o navegador esteja aplicando os padrões corretamente. Caso queira garantir que seu layout fique o mais parecido possível em qualquer dispositivo / navegador, é interessante o uso da técnica de "reset.css", onde é aplicado definições padrões mesmo que estas já sejam contempladas pelo browser.
Answer: Vou te dar um exemplo simples, todo elemento precisa de um mínimo para ao menos existir, e umas dessas propriedades básicas é o display, todos os elementos tem algum tipo de display, caso contrário ele seria display:none e nem na tela ficaria visível pra vc. Como é o casa das tags


On the canonical morphism $B\otimes_A N \to N$?

Question: Let  $A$  and $B$ be two commutative rings with unit.    Let $\rho: A \rightarrow B$ be a ring homomorphism and $N$ a $B$-module (and thereby a $A$-module).
Question: Is there a canonical morphism $B\otimes_A N \to N$? If so, what is the expression of this morphism?
Comment: $N$ is a $B$-module. So there is a scalar multiplication map, and it's bilinear…
Answer: Consider the $A$-bilinear map:
$$\begin{align*} B\times N\to N \\ b\times n \to bn. \end{align*}$$
By the universal property, this induces a unique $A$-module (surjective) homomorphism  $$B\otimes_{A} N\to N.$$
Comment: Your inverse map is not correct unless the tensor product is with respect to $R$, which is not the case in the question (also as a general rule try not to change the notations used in the question).
Comment: I didn't pay attention that the tensor product being over $A$, I will edit my answer.


Local Inversion theorem (first sentence of the proof)

Question: In this local inversion theorem I want to prove that a map f between two Banach spaces E and F is a local diffeomorphism.
In the proof it says :
Without loss of generality we can consider the case where E= F.
Sorry for this simple question, but why is that? How does this imply the general case where f goes from an open set U in E to F?
In finite dimensions we could probably have an embedding of E in F or something similar...but I'm not sure how to justify this for general Banach spaces...
Comment: What is the sketch of the proof? This would help in determining why there is no loss of generality.
Comment: There are 4 main steps:   1) Simplification of the problem ( we take E= F, and a=b=0 the point where df_a is invertible, df_a = Id_E)  this is possible because translations and multiplication by df_a are also diffeomorphisms and the translated of an open is an open set. 2) Existence of local reciprocal map 3) f^{-1} is Lipchitz continuous 4) f^{-1} is class C1.    So the first step helps for the other steps.
Answer: Just so we're on the same page, regarding notation etc I'll state the IFT:

Inverse Function Theorem:
Let $E,F$ be real Banach spaces, $f:E \to F$ be a $C^k$ map such that at a point $x_0 \in E$, the differential $df_{x_0}$ is an invertible element of $L(E,F)$. Then, there is an open set $U \subset E$ containing $x_0$, an open set $V \subset F$ containing $f(x_0)$, such that $f:U \to V$ is invertible. The inverse map $f^{-1}:V \to U$ is also $C^k$,
.... (now they give a formula for the derivative of the inverse)

Suppose now that we manage to prove this theorem in the case where the domain and target space of the function are both $E$. We are now going to deduce it in the general case as follows: with notation as in the theorem, define a new map
\begin{equation}
g = (df_{x_0})^{-1} \circ f
\end{equation}
Notice that it maps $E$ into $E$. Since $f$ is $C^k$, and $(df_{x_0})^{-1}$ is an element of $L(F,E)$ , it is $C^{\infty}$. Hence $g$ being the composition is atleast $C^k$. A simple computation shows (use chain rule)
\begin{align}
dg_{x_0} = \text{id}_E \tag{*}
\end{align}
So, $dg_{x_0}$ is an invertible element of $L(E,E)$. Now all the relevant hypotheses on $g$ are satisfied. So, by the special case, we know that there an open set $U \subset E$ containing $x_0$, an open subset $W$ containing $g(x_0)$, such that $g: U \to W$ is $C^k,$ with $C^k$ inverse $g^{-1}: W \to U$.
Now notice that by definition, $f = df_{x_0} \circ g$. So, if we take $V = f[U]$, then $f:U \to V$ is a composition of invertible maps, it is also invertible, with $f^{-1} = g^{-1} \circ (df_{x_0})^{-1}$, which is also a composition of $C^k$ maps, and hence $f^{-1}$ is $C^k$. This proves the simplification is sufficient.

Extra Remarks:

Notice that by $(*)$, we may also assume that $df_{x_0} = \text{id}_E$.
One result which I implicitly made use of above is Banach's Isomorphism theorem, which states that if $T: E \to F$ is a linear and continuous (equivalently, bounded) map (which by definition is what it means to be an element of $L(E,F)$) which is invertible, then the inverse $T^{-1}:F \to E$ is guranteed to be continuous (equivalently bounded), so that $T^{-1}$ is in $L(F,E)$. Continuity and linearity then immediately implies both are $C^{\infty}$. In finite dimensions this is obvious, because every linear map between finite-dimensional spaces is continuous, but apparently it's a hard theorem in infinite dimensions.
Comment: Thanks a lot for such a great answer. Just what I was looking for. Now I finally have a neat proof! Yes the Banach isomorphism theorem, as it's linked to Baire's theorem, is not obvious in infinite dimensions. Ok so you use it k times for the class Ck case.  Apparently one could also use the (easier) Neumann series to prove that $ df_{x_0}^{-1}$ is continuous


Como ativar uma class no botao html atraves do Javascript

Question: Tenho um botão PAUSE/PLAY e queria que ao clicar no PAUSE fosse ativada a class .carousel-control-next-icon.
Aqui o trecho do código que quero inbutir a class: .carousel-control-next-icon ao ser clicada.

Tentei fazer usando onclick, mas não consegui.

      #player{
        position: absolute;
        bottom: 40%;
        right: 40px;
        z-index: 999;
      }
      .controls i{
        background: linear-gradient(180deg,#fff 0%,#ddd 100%);
        padding: 11px 13px 11px 13px;
        border-radius: 50%;
      }
      .controls i{
        font-size: 20px;
        color: #178F5D;
        text-align: center;
        transition: 1s;
        transition: 100ms all ease;
      }
      .controls label {
        position: absolute;
        cursor: pointer;
      }
      #myBtn,
      #myBtn2 {
        display: none;
      }
      #play:checked ~ .controls #myBtn,
      #pause:checked ~ .controls #myBtn2 {
        display: block;
      }
      [name="player"] {
        display: none;
      }
<link href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet"/>

 <input type="radio" name="player" id="pause" checked>
 <input type="radio" name="player" id="play">

Comment: Você quer atribuir a classe `.carousel-control-next-icon` no `` ou no ``?
Comment: @CmteCardeal quero atribuir no label id=myBtn2 porq se colocar no i acho q não da certo
Answer:

  var btn2 = document.getElementById('myBtn2')
  var showTag = document.querySelector('.controls') //pegar a div com class 'controls' sóp ara testarmos no final
  btn2.addEventListener('click', addClass)//Event Listener de click o mesmo evento de 'onclick' na tag html, (ao clicar executar a função addClass)

  function addClass() {
    if (btn2.getAttribute('class') == '.carousel-control-next-icon') {
     btn2.setAttribute('class', '.carousel-control-next-icon')
    } // Vai pegar/checar se o valor atributo class do label btn2 é igual a .carousel-control-next-icon, e se for ele apenas vai substituir pelo mesmo e não adicionar mais class
    else {
      btn2.setAttribute('class', '.carousel-control-next-icon')
    } //Caso a class do label btn2 não for igual a .carousel-cont... ele vai adicionar uma class. Adicionando na tag label um class=".carousel-control.next-icon"
    alert(showTag.innerHTML)// Só Para testar e conferir se a class está no label
  }

          #player{
            position: absolute;
            bottom: 40%;
            right: 40px;
            z-index: 999;
          }
          .controls i{
            background: linear-gradient(180deg,#fff 0%,#ddd 100%);
            padding: 11px 13px 11px 13px;
            border-radius: 50%;
          }
          .controls i{
            font-size: 20px;
            color: #178F5D;
            text-align: center;
            transition: 1s;
            transition: 100ms all ease;
          }
          .controls label {
            position: absolute;
            cursor: pointer;
          }
          #myBtn,
          #myBtn2 {
            display: none;
          }
          #play:checked ~ .controls #myBtn,
          #pause:checked ~ .controls #myBtn2 {
            display: block;
          }
          [name="player"] {
            display: none;
          }
    <link href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet"/>

     <input type="radio" name="player" id="pause" checked>
     <input type="radio" name="player" id="play">

E caso queira ficar Ativando e Desativando a class (apagando e criando-a) basta apenas colocar um removeAttribute('class')
Exemplo:
OBS: Que caso queira ficar desativando e ativando a class do <label id=myBtn2 e ativar apenas quando clica nele, você vai ter que colocar um onclick em html ou um enventListener em javascript para chamar a função addClass para o outro label o <label id=myBtn desativar a class do myBtn2

  var btn2 = document.getElementById('myBtn2')
  var showTag = document.querySelector('.controls') //pegar a div com class 'controls' sóp ara testarmos no final
  btn2.addEventListener('click', addClass)//Event Listener de click o mesmo evento de 'onclick' na tag html, (ao clicar executar a função addClass)

  function addClass() {
    if (btn2.getAttribute('class') == '.carousel-control-next-icon') {
     btn2.removeAttribute('class')
    } // Vai apagar a class se o valor atributo class do label btn2 é igual a .carousel-control-next-icon, e se for ele apenas vai substituir pelo mesmo e não adicionar mais class
    else {
      btn2.setAttribute('class', '.carousel-control-next-icon')
    } //Caso a class do label btn2 não for igual a .carousel-cont... ele vai adicionar uma class. Adicionando na tag label um class=".carousel-control.next-icon"
    alert(showTag.innerHTML)// Só Para testar e conferir se a class está no label
  }

          #player{
            position: absolute;
            bottom: 40%;
            right: 40px;
            z-index: 999;
          }
          .controls i{
            background: linear-gradient(180deg,#fff 0%,#ddd 100%);
            padding: 11px 13px 11px 13px;
            border-radius: 50%;
          }
          .controls i{
            font-size: 20px;
            color: #178F5D;
            text-align: center;
            transition: 1s;
            transition: 100ms all ease;
          }
          .controls label {
            position: absolute;
            cursor: pointer;
          }
          #myBtn,
          #myBtn2 {
            display: none;
          }
          #play:checked ~ .controls #myBtn,
          #pause:checked ~ .controls #myBtn2 {
            display: block;
          }
          [name="player"] {
            display: none;
          }
    <link href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet"/>

     <input type="radio" name="player" id="pause" checked>
     <input type="radio" name="player" id="play">

Espero ter ajudado


Email verification programmatically

Question: I have a custom registration form with three options(premium, trial and group).
The premium user does not need email verification but other two do. So I need to send verification email pro-grammatically.
I have searched google where I got only options for some modules. But my need is only sending an email with unique URL for verifcation.
Please suggest me the drupal function that helps me achieve this.
Thank you
Answer: You could use hook_mail_alter to abort the verification email being sent in the condition that the user is using the premium option. As stated on the api.drupal.org page if you set the 'send' parameter of the $message data to FALSE it will abort sending the email.
You'll need to retrieve the 'premium' context within your implementation of hook_mail_alter() to achieve this, which may require setting that property on the user, the session, or in a custom database table.

Email verification programmatically


Leap day removal

Question: My Dataset contains two columns, one with Date as YYYYMMDD and a column with another value.
The date column is type Date, as I used function as.Date to transform the values from int to character to Date %Y%m%d.
I am using daily data, therefore, I have to delete the leap days. I tried many things, but I did not find anything that works yet. I want to remove the 29th of February for each leap year (i.e. 1992, 1996, 2000, 2004, 2008, 2012, 2016, 2020). How can I do this?
dsKNMI$Date <- as.character(dsKNMI$Date)
dsKNMI$Date <- as.Date(dsKNMI$Date, format = "%Y%m%d")

Dates are in format 1990-12-31
Answer: With Base R,
dsKNMI <- dsKNMI[!(format(dsKNMI$Date, format = "%m") =="02" &
                      format(dsKNMI$Date, format = "%d")=="29"),]

Leap day removal


Rails facebook app returns 422 " the change u wanted was rejected" error

Question: My facebook is just approved today. However,when I clicked on "Go to app", it returns Rail 422 " The change you wanted was rejected. error" I set my Canvas URL to my site url and I am using Omniauth, so I am just wondering what did I miss.
Answer: Try removing
protect_from_forgery

In the application_controller, or
# you can disable csrf protection on controller-by-controller basis:
skip_before_filter :verify_authenticity_token

Facebook uses signed_request that you can decode and use to verify that the request actually comes from facebook, so protect_from_forgery is unnecessary for applications that are accessible through the facebook canvas only.
More here: Canvas Authentication & ActionController::RequestForgeryProtection
Answer: Facebook's omniauth error "the change you wanted was rejected"
might appear because of your validations set in the model. I had to refactor my validation for users having one unique email, which wasn't working when a user would try to facebook login with the same email.
Look at your logs. heroku logs -t


Thinking that all individuals pursue "selfish" interest is equivalent to assuming that all random variables have zero covariance

Question: Might be off-topic here but on-topic on Math, Eco or Philo SE. Pls migrate if needed.
I read that: Thinking that all individuals pursue "selfish" interest is equivalent to assuming that all random variables have zero covariance. -- Nero
What does that mean, and what does it have to do with random variables and covariance?
There is an explanation: I read it as saying that people have many interests in common, so pursuing "selfish" interests can also be altruistic to some extent.
Another: That some people do in fact work towards the common good, or conversely, are outright malevolent rather than focused on personal gain
Assuming those are the meanings, I don't see what that has to do with random variables and covariance.
My guess: Given an index I, we list all random variables conceivable: $\{X_i\}_{i \in I}$.
It is clear that $Cov(X_j, X_k) \ \forall j, k \in I$, if well-defined, may or may not be zero.
Is covariance among two random variables an analogy for common interest among two people?
Comment: Just read on modern interpretations of Adam Smith's Wealth of Nations work. It helps to survey what's been discussed so far before venturing into this kind of ontology.
Comment: @Aksakal The point is to relate it back to covariance and random variables...?
Comment: Covariance has very little to do with this. As long as there's some randomness the argument will work.
Answer: Short Answer: No, the analogy doesn't hold up well.
Longer answer:
The analogy would be astute if and only if interests among people were random- they aren't.
Human beings have universal needs along several axes- physical, emotional, arguably spiritual.
This leads me to conclude that in a hypothetical space of all possible interest $s$, there could be defined some directed function $f$ to describe all practical human interest space, and that this function would have some $x$-dimensional vector tendency within space $s$ and therefore human interests would co-vary along those vector tendencies.

It would also follow that any interest pursued according to those vector tendencies would correlate with other human interests-

In effect, self-interested but human interests correlate with altruistic but human interests in the hypothetical geometric space of all possible interests. That is, the controlling property is the humanness of the interest.
The random variables you mention in the space of all possible covariances have no identifiable similar $x$-dimensional vector tendency, certainly not to the same degree.

I have no doubt that a philosophical answer would look very different indeed.
Comment: This answer conflates the colloquial definition of random, which approximates the definition of uniformly random, with the definition of a random variable, which is a more general concept.
Comment: Thanks Thomas Cleberg! ^-^ Is it against SE rules if I were to cross-post given your last line?
Comment: If I were you, if you're interested in the philosophical answer to the question, I would try to formulate the question differently as a philosophical question and then ask that differently formulated question in philo SE
Comment: Other than that, it's a fantastic answer by the way
Comment: One thing I'd add though is that if your model could separate account for the reasons outside costly signaling, kinship, reciprocity, reputational benefit, group selection etc, then the residuals should be uncorrelated if none of those processes were in effect. Of course, that separation is basically impossible.


Building EntityForm in Drupal 8

Question: I'm continuously getting the error: Drupal\Core\Config\Schema\SchemaIncompleteException: Incomplete or missing schema for mymodule.myentity.myentityname in Drupal\Core\Config\Entity\ConfigEntityBase->toArray() (line 248 of core/lib/Drupal/Core/Config/Entity/ConfigEntityBase.php).
Here is how I'd implemented the EntityForm:
class MymoduleForm extends EntityForm {
  public function form(array $form, array &$form_state) {
    $form = parent::form($form, $form_state);
    $entity = $form_state['entity'] = $this->getEntity();
    $form['name'] = array(
      '#type' => 'textfield',
      '#title' => t('Name'),
      '#default_value' => empty($entity->name) ? '' : $entity->name,
      '#required' => TRUE,
    );
    ...
    ...

    return $form;
  }
  public function validate(array $form, array &$form_state) {
    parent::validate($form, $form_state);
    $entity = $this->getEntity();
    //perform validation
  }
  public function save(array $form, array &$form_state) {
    $entity = $this->getEntity();
    $entity->save();
    print_r($entity);
    //throws error here

    $edit_link = \Drupal::linkGenerator()->generateFromUrl($this->t('Edit'), $this->entity->urlInfo());
  }

Doing a print_r($entity) shows two Object results: one showing all the values I'd filed up in the form and the other completely emplty.
Answer: That error has nothing to do with your form.
You need to provide a config schema for your config entity, see examples in node.schema.yml and many other places.
See also the configuration schema documetation.
Comment: Oh wait, there's a fully-fledged config entity example in the [Examples module](https://www.drupal.org/project/examples), that should show you what's needed
Comment: @con_28 Giving a specific example isn't really possible as we don't know what fields your entity has. D8 is in alpha, so tutorials will be hard to come by. See `node.type.*` in core/modules/node/config/schema/node.schema.yml, `comment.type.*` in core/modules/comment/config/schema/comment.schema.yml for examples, after reading the docs Berdir linked to
Comment: Could provide an example? or a Tutorial link? I'm quite new to D8.
Comment: @Clive: Thankyou. That was helpful.

Building EntityForm in Drupal 8


EM algorithm to estimate discrete Markov chain transition probabilities

Question: I'm surveying algorithms to estimate the transition probabilities of a discrete Markov chain and I found that EM approach could used. However I am not able to find any simple description on how to implement the EM algorithm to fit a discrete Markov chain giving a data sequence. Is there any R function that does this?
Comment: I can help you if you can explain the problem a bit better. Since you wish to use EM, there should be an observed (known) component and an unobserved (hidden) component. In your case, what is observed and what is latent/hidden? The mention of EM and Markov chains collectively bring to mind the Baum-Welch algorithm for Hidden Markov Models. If that is the case, there is a package called HMM in CRAN.
Answer: you should check "Estimation of the transition matrix of a discrete-time Markov chain" by Bruce A. Craig. This article describe how to do what you want.
wish this helps
Answer: Before you proceed with any algorithms for hidden Markov models please read one of the keystone papers in HMMs:
http://www.cs.ubc.ca/~murphyk/Bayes/rabiner.pdf
In it you will find a wonderful overview of HMMs as well as 3 key algorithms to solve the 3 problems in HMMs that are usually of interest!
Also there are several implementations of HMMs in R:
Here is one http://cran.r-project.org/web/packages/HiddenMarkov/HiddenMarkov.pdf


iphone add framework

Question: I have added CFNetwork.framework,but when I import the header file "CFNetwork.h",it said"CFNetwork.h:no such file or directory"
Answer: One of these will work: Import the top level header instead.
<code>#import <CoreServices/CoreServices.h>
</code>
Or
<code>#import <CFNetwork/CFNetwork.h>
</code>

iphone add framework


Como parar um loop Each javascript

Question: Olá, estou desenvolvendo um mecanismo de busca num rank para bot na twitch.tv segue o codigo abaixo
const cheerio = require('cheerio')
const URL ='https://shadowarena.pearlabyss.com/en-US/Arena?battleType=0&server=sa'
let rankYoda;
async function acesso(){
    const response = await request(URL)
    const $ = cheerio.load(response)
    $('.box_list_area').each((i, e) => {
        const name = $(e).find('.thum_name').text()
        const rank = $(e).find('.thum_rank').text()
        if(name === "YoDaSL"){
            rankYoda = rank
        }

        }
        )}
acesso()
console.log(rankYoda)

Sempre que rodo, ele me retorna no console o valor undefined, e eu gostaria que quando tal nome "YoDaSL" seja encontrado, parasse o loop e associasse a uma variavel fora do loop, para poder dar console.log nela, alguém poderia me ajudar?
Comment: Olá! Vc declarou a variável `let rankYoda;` antes da função assíncrona. Até aí tudo bem. Logo após a função vc executou a função e fez um console.log pra ver o valor da variável. O valor da variável é undefined pq a função onde vc quer atribuir um valor a ela é assíncrona, ou seja, não é no mesmo momento que o JS é executado. Faz um teste de olho: olhe pro seu código de cima pra baixo em uma fração se segundo. A função `acesso()` já retornou alguma coisa? Com certeza não, pq ela é assíncrona, ou seja, não respeita o tempo, depende do retorno.
Comment: Entendi, mas como eu poderia fazer isso então? Sou bem iniciante em javascript, dei umas pesquisadas mas não encontrei nada...
Comment: Use a lógica e a criatividade.
Answer: Você está usando funções assíncronas, isso significa que o javascript roda a sua função "paralelamente" ao restante do código e você está tentando trabalhar acessar de forma síncrona.
Funções assíncronas executam com callbacks, são funções que executam quando a função principal terminar, exemplo:

const URL ='https://httpbin.org/get';
$.get(URL, r => console.log(r));
console.log("Meu log");

Perceba que ao executar o snippet, mesmo a função $.get executando primeiro, "Meu log" é o primeiro a aparecer no console. Isso acontece porque o javascript só executa o console.log(r) quando a requisição estiver terminada.
Na sua função então eu sugiro as seguintes mudanças:
const cheerio = require('cheerio')
const URL ='https://shadowarena.pearlabyss.com/en-US/Arena?battleType=0&server=sa'
let rankYoda;
// já que está usando "await" no request e retornando o valor isso disponibilizará os dados para usarmos na outra função
async function requestData(){
    const response = await request(URL)
    const $ = cheerio.load(response);
    return $;
}
// aqui os dados carregados pelo cheerio estarão no parâmetro $
function searchUserRank(user, $){
    $('.box_list_area').each((i, e) => {
        const name = $(e).find('.thum_name').text()
        const rank = $(e).find('.thum_rank').text()
        if(name === user){
            return rank;
        }

    });
};
cheerioData = requestData();
rankYoda = searchUserRank("YoDaSL", cheerioData);
console.log(rankYoda);

Não testei mas acredito que seja algo próximo a isso que você está procurando
Comment: Opa mano, obrigado pela ajuda! Testei o seu codigo, e está retornando o seguinte erro: `$('.box_list_area').each((i, e) => {
    ^

TypeError: $ is not a funnctio` Mesmo assim, obrigado pelo norte!

Como parar um loop Each javascript


order isomorphisms of $[0,1]^n$

Question: An order isomorphism is any bijective map $f:X\rightarrow Y$ between posets such that $f(x)\leq f(y)$ if and only if $x\leq y$.
Let $[0,1]^n$ be a poset with the standard order: $(x_i)\leq (y_i)$ if and only if $x_i\leq y_i$ for all $i$.
If $\sigma\in S^n$ is a permutation then the map $g_\sigma:[0,1]^n \rightarrow [0,1]^n$ defined by $g_\sigma((x_i)) = (x_{\sigma i})$ is obviously an order isomorphism.
If we have $n$ strictly increasing bijective maps $f_i:[0,1]\rightarrow [0,1]$ then the product map $f=(f_i):[0,1]^n\rightarrow [0,1]^n$ is also an order isomorphism.
My question is, can any order isomorphism of $[0,1]^n$ be written as a composition of these types of order isomorphisms (this is obviously true for $n=1$)? (I'm not sure if we need to assume continuity as well or if that automatically follows from the map being an order isomorphism.)
It is not too hard to prove that if the order isomorphism is linear that it must be given by some $\sigma\in S^n$, so if for any order isomorphism we can find a product map such that the composition is linear we would be done, but I'm not sure if this can be done.
Any more general references on where I can find answers to questions similar to these would also be appreciated.
Answer: My question is, can any order isomorphism of $[0,1]^n$
be written as a composition of these types of order isomorphisms?
Yes.
Call the first type of map
a coordinate permutation and the second type
a product automorphism.
Let $f\colon [0,1]^n\to [0,1]^n$ be a poset automorphism. It must also
be a lattice automorphism, since the lattice operations are definable
order-theoretically. The key fact you need about the lattice $[0,1]^n$ is this:
The nonzero join irreducible elements of $[0,1]^n$ are the tuples
that are nonzero in exactly one coordinate.
In particular, the tuple ${\bf e}_i$ that is $1$ in the $i$th
coordinate and zero in all others
is a maximal join irreducible and the interval $[{\bf 0},{\bf e}_i]$
is a maximal chain of join irreducibles. Every tuple $(r_1,\ldots,r_n)$
is a join of at most $n$ join irreducibles, namely
$(r_1,0,\ldots,0), (0, r_2,\ldots,0), \ldots$.
Since the finite set $E=\{{\bf e}_1, \ldots, {\bf e}_n\}$
is definable order theoretically as the set of maximal join irreducibles,
it must be invariant
under $f$. There is a coordinate permutation $\sigma$
that agrees with $f$ on $E$, hence $g:=\sigma^{-1}\circ f$
is the identity on $E$. The endpoints of the intervals
$$
[{\bf 0},{\bf e}_i] = \{0\}\times \cdots\times [0,1]\times \cdots \times \{0\}
$$
are fixed by the automorphism $g$,
so $g$ restricts to an automorphism of
each of these intervals.
Let $g_i\colon [0,1]\to [0,1]$ be taken so that
$g(0,\ldots,x,\ldots,0)=(0,\ldots,g_i(x),\ldots,0)$.
Then $G = \prod g_i$ is a product automorphism which
agrees with $g$ on any interval of the form
$[{\bf 0},{\bf e}_i]$. That is,
$G^{-1}\circ g = G^{-1}\circ \sigma^{-1}\circ f$
fixes every elements of every interval of the form
$[{\bf 0},{\bf e}_i]$. But now
the order automorphism $G^{-1}\circ \sigma^{-1}\circ f$
fixes every element of $[0,1]^n$,
since every element of this lattice is a join
of fixed points of
$G^{-1}\circ \sigma^{-1}\circ f$. This shows that
$G^{-1}\circ \sigma^{-1}\circ f(x) = x$, or
$f(x) = (\sigma\circ G)(x) = (\sigma\circ (\prod g_i))(x)$.

order isomorphisms of $[0,1]^n$


Why is logistic regression well calibrated, and how to ruin its calibration?

Question: In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression.
Why is logistic regression well calibrated? How could one ruin the calibration of a logistic regression (not that one would ever want to - just as an exercise)?
Answer: Although this question and its first answer seems to be focused on theoretical issues of logistic regression model calibration, the issue of:

How could one ruin the calibration of a logistic regression...?

deserves some attention with respect to real-world applications, for future readers of this page. We shouldn't forget that the logistic regression model has to be well specified, and that this issue can be particularly troublesome for logistic regression.
First, if the log-odds of class membership is not linearly related to the predictors included in the model then it will not be well calibrated. Harrell's chapter 10 on Binary Logistic Regression devotes about 20 pages to "Assessment of Model Fit" so that one can take advantage of the "asymptotic unbiasedness of the maximum likelihood estimator," as @whuber put it, in practice.
Second, model specification is a particular issue in logistic regression, as it has an inherent omitted variable bias that can be surprising to those with a background in ordinary linear regression. As that page puts it:

Omitted variables will bias the coefficients on included variables even if the omitted variables are uncorrelated with the included variables.

That page also has a useful explanation of why this behavior is to be expected, with a theoretical explanation for related, analytically tractable, probit models. So unless you know that you have included all predictors related to class membership, you might run into dangers of misspecification and poor calibration in practice.
With respect to model specification, it's quite possible that tree-based methods like random forest, which do not assume linearity over an entire range of predictor values and inherently provide the possibility of finding and including interactions among predictors, will end up with a better-calibrated model in practice than a logistic regression model that does not take interaction terms or non-linearity sufficiently into account. With respect to omitted-variable bias, it's not clear to me whether any method for evaluating class-membership probabilities can deal with that issue adequately.
Answer: Logistic regression is a classification method that basically learns a probability function $\pi_\theta(x)$ over the input space by fitting the parameters $\theta$. If the predicted probabilities are learned with the appropriate loss function, than logistic regression has the potential to learn an unbiased estimation of the binary event probabilities, whenever its has sufficient capacity (input features).
The log loss allows such unbiased estimation. Consider the fact that the log loss function is simply the negative log likelihood of a Bernoulli distribution $z \thicksim \text{Ber}(p)$. The maximum likelihood estimation for $p$ is unbiased given a set of observations for variable $z$. In the case of classification over some input space $\mathcal{X}$, one can imagine having one Bernoulli distribution for all points in $\mathcal{X}$. Most often, you will only have 1 observation $y_i$ per Bernoulli distribution, which is located at $x_i$. Jointly applying maximum likelihood estimation for all observed Bernoulli distributions $y_i \thicksim \text{Ber}(\pi(x_i))$ will apply several constraints to $\pi_\theta$. Since all these constraints leads to unbiased estimations, and as long as the function $\pi_\theta$ is sufficently flexible to fit the true underlying probability function $\pi^*$, then the learning procedure is consistent and will converge to the optimal model as you get more data. Thus, limiting the model capacity (fewer features for instance) can hinder the calibration of a logistic regression by increasing the distance between the best learnable model and the true model.
Using an incorrect observation model with the logistic regression will lead to uncalibrated probabilities. Modeling binary events with a normal distribution is inappropriate, and should not be used in combination with logistic regression. The loss function corresponding to the normal distribution observation model is the Mean Squared Error. Thus, using an MSE loss would certaintly hinder its calibration.
Comment: The scikit-learn quotation about "optimizes log loss" isn't an effective explanation, because there is no necessary connection between this and being unbiased.  Unless I'm mistaken, the correct answer to the question will need to invoke the *asymptotic unbiasedness* of the *maximum likelihood estimator* typically used in logistic regression procedures.
Comment: Careful calling logistic regression a classification method on this site! Thank you for the answer - it seems you are implying that the log loss objective is the reason for calibration (assuming the model is adequately flexible)?
Comment: A follow up - you say calibration requires unbiased estimation of the probability - hence penalization ruins calibration?
Comment: By definition, penalization or regularization, is a bias injection which often seeks to reduce the variance of the estimator. A massive regularization can dominate the data part of the objective function, and definitely ruin calibration.
Comment: « LogisticRegression returns well calibrated predictions by default as it directly optimizes log-loss »
- https://scikit-learn.org/stable/modules/calibration.html
Comment: @whuber isn't the asymptotics needed to claim Max Likelihood estimates of the coefficients are unbiased, not the probability estimates which is being discussed in this question?


How to efficiently expand a simple random sample after population growth?

Question: I have a simple random sample (SRS) $S$ of size $n$ drawn from a population $D_1$ containing $N_1$ members.  A new set $D_2$ with $N_2$ members is added to the population.  I would like to create a new set $S'$ that is a SRS from $D_1 \cup D_2$.
One strategy that won't work is to draw a SRS of size $nN_2/N_1$ (or any other size) from $D_2$ and add is to $S$. (The result would be a stratified random sample from $D1 \cup D2$, but not a SRS from it.)
A strategy that will work is to first make one trial on a random variable with a hypergeometric distribution (population size $N_1+N_2$, number of draws $n'$, and number of success states in the population $N_1$).  The resulting value $n'_{1}$ is the number of items in $S'$ that should be taken from $D_1$.  If $n'_{1} < n$, we discard a SRS of size $n - n'_{1}$ from $S$ to form the initial $S'$.  If $n'_{1} > n$, we select an additional SRS of size $n'_{1} - n$ from $D_1 \setminus S$, and put these items, plus all items from $S$ in $S'$.  We also draw an SRS of size $n'_{2} = n' - n'_{1}$ from $D_2$ and add it to $S'$.
I'm interested in choosing $n'$ to minimize $E[|S' \setminus S|]$, the expected size of $S' \setminus S$.  In other words, the expected value of the number of items from $D_2$ plus the number of new items, if any, from $D_1$.
I have three questions:
Q1. If we have the additional constraint that $n' \ge j$ for some $j$, is there a simple argument that the optimal $n'$ is $j$?  Seems like it has to be, but I'd rather not work out the expected value formula and slog through the derivatives.
Q2. Does anyone know of a better algorithm for the above problem?
Q3. The problem I'd really like to solve is more complex. Each item in $D_1$ and $D_2$ is either an A or a B.  $S$ has at least $k$ A's, and I want to minimize $E[|S' \setminus S|]$ subject to the constraint that $S'$ has at least $k$ A's.  The above algorithm no longer works, because we can't pick $n'$ in advance. (We don't know how rich $D_2$ is in A's, and we might get unlucky anyway.)  Is there a simple algorithm for this case?  I've perused
Thompson, S. K. and Seber, G. A. G.  Adaptive Sampling. Wiley, New York, 1996
but can't find exactly this problem treated.  But my gut tells me that both the above problems have been faced and solved many times before.
Comment: Clever, and I certainly can't think of a better way of doing this.  But for Q1 to make sense we need another constraint; otherwise the smaller $n'$ gets the better it meets your criterion, with $n'=0$ meaning you don't have do any new sampling at all.
Comment: There are some slight tweeks: for example if $n'_{1} \le n/2$, it may be faster to take an SRS size $n'_{1}$ from $S$ than to discard an SRS size $n-n'_{1}$.
Comment: Peter - Good point about $n'$.  I had simplified my actual problem too much. I've edited my question above to give the full problem.
Comment: Henry - Good point, though I'm actually more interested in a simple algorithm than an efficient one in this case. Fast existing software will do the sampling, but the overall process needs to be comprehensible to non-statisticians.
Comment: Indeed, from the standpoint of simplicity, minimizing the number of times a person calls the simple random sampling capability of an existing piece of software is probably the main factor.
Comment: With Q1, if you add any constrains, you no longer have SRS. If you have constraints like that, it is quota sampling, which is rarely, if ever, any good.
Answer: You are right, there's been some thought put into how to sample from growing populations by sampling statisticians. I think you would want to look at the work of Larry Ernst of the US Bureau of Labor Statistics: http://www.google.com/search?q=lawrence+ernst+bls+sampling+overlap. As I am still not sure what your statistical problems is -- estimating the fraction or the number of As in the total population? -- I cannot recommend anything specific, but I expect that you will find something suitable from the body of work he develops and cites. Most statistical solutions would involve stratified samples though; these folks are interested in developing statistical solutions in which they can quantify the probabilities of selections of the individual units and the pairs of units. The issue of how the algorithm scales or runs in real time is rarely of interest, as sample is to be taken once, so it's OK if it takes a long time. For you, apparently, timing is an issue.
Comment: Permanent random numbers are available to my users, so I'm going to declare this answered - thanks!
Comment: StasK - Thanks for the pointer.  The literature is much larger than I expected!  The key terms are ones like "sample overlap",  "sample coordination", and "synchronized sampling".  As you observe, my problem is much simpler than the ones the literature addresses, so I'm having to dig back to the basics but I'm sure I'll find it.  Drawing a sample in order of "permanent random numbers" (essentially hash codes) provides one easy solution, but I'm not sure the software platforms my users have support that capability.


Quantity of binary arrays of length n with k consecutive 1

Question: Input: N = 4, K = 3
Output: 3 // "111", "1110", "1111"

I solved this with this code:
function getQuantity(n, k) {
  const nums = [];
  const re = new RegExp(`^.*[1]{${k}}.*?$`, "m");

  // Convert to decimal
  var maxDecimal = Math.pow(2, n) - 1;

  // For every number between 0->decimal
  for (let i = 0; i <= maxDecimal; i++) {
    if (re.test(i.toString(2))) {
      nums.push(i.toString(2));
    }
  }

  return nums.length;
}

But I suppose this is overhead solution. Maybe there is better way to solve this?
Comment: Seems like a case for https://codereview.stackexchange.com/
Comment: Is you your question Find number of binary number representation of length N containing K consecutive 1
Comment: @ShubhamSrivastava yes! I just need the quantity of these numbers, because of it I suppose my solution is overhead
Comment: you might be able to solve this with permutation, I am on phone right now so i cannot write up a script
Comment: If your code is working, even too slow, it may help if you provide more results, for example k = 3 and n = 4, 5, 6, 7, 8, 9, 10. This will allow us to validate other methods, without the need to implement your current method. Issue is that I don't know javascript, but I may provide another algorithm anyway. I may not be alone in this case
Comment: @Damien I'm not so good in algoritms. I just thought that there is some way to get the quantity without generating and looping through whole list of binaries(for example for n=10) I got 1024 iterations. So this O(2^N) complexity which is quite bad.
Comment: My concern was not the complexity of your method, but the result ! For example, you mention 3 arrays for (4, 3). And for (10, 3) ? I get 520 with a faster method, but I am not sure it is the correct result
Comment: Here is results:
N=4, K=3 -> 3 // "111", "1110", "1111"
N=5, K=4 -> 8 // "111", "1110", "1111", "10111", "11100", "11101", "11110", "11111"  N=5, K=4 -> 3 // "1111", "11110", "11111"
Comment: @Damien yeah! I also get 520 for (10, 3)
Answer: Your method has a complexity O(2^N). Using a recursive or iterative method gives a O(N) complexity.
The relation is this one: f(n+1, k) = 2*f(n, k) + 2^{n-k} - f (n-k, k)
with f(n, k) = 0 if (n < k) and f(k, k) = 1
The last term 2^{n-k} - f (n-1-k) corresponds to the additional number of new solutions,
when a 1 is added in the (n+1)th position, with one 0 and (k-1) 1 just before. Then there are n-k bits that remain to be set, i.e. potentially 2^{n-k} new solutions. However, the term - f(n-k, k) is needed not to count solutions already included in the f(n, k) term.