Solr search for distinct "nids" based on keywords

Question:  <fields>
    <field name="mongo_id" type="string" indexed="true" stored="true" required="true" />
    <field name="nid" type="int" indexed="true" stored="true" required="true" />
    <field name="keywords" type="text_general" indexed="true" stored="false" />
 </fields>

I want to get back ALL of the nid's (distinct) that match the keywords.
 $solr = new Apache_Solr_Service(SOLR_HOST, SOLR_PORT, SOLR_WEBAPP);

 //How would I search here?
 $results = $solr->search('search', 0, 100);

EDIT:
Does this look right?
    $solr = new Apache_Solr_Service(SOLR_HOST, SOLR_PORT, SOLR_WEBAPP);
    $results = $solr->search(Apache_Solr_Service::escapePhrase($_GET['keywords']), 0, 0, array('facet' => 'true', 'facet.field' => 'nid', 'rows' => 0, 'facet.mincount' => 1));

    foreach($results->facet_counts->facet_fields->nid as $nid=>$count)
    {
        $nids[] = (int)$nid;
    }

Answer: I think you'll want to use faceting. Facet on the nid:
?q=search&facet=true&facet.field=nid&rows=0

It will give you a list of the nids that match along with the number of times each is found in the search. With &rows=0 only the facet counts will be returned.
Edit: One more thing, make sure to set facet.limit=-1 (or any other negative number) to specify ALL results, and facet.mincount=1 to make sure that only matches are returned. Otherwise it will return non-matches with counts of zero. If you want to sort by count (it won't by default if facet.limit is less than 1) also add facet.sort=count.
Comment: I've not used the PHP Solr Service so I couldn't say for certain, but to my untrained eyes it looks OK.


If the Bayesian probability is not a belief, what is it?

Question: In this blog by William Briggs, who seems to be  a prolific lecturer/writer on pop-statistics, he condemns the "Bayesian Metaphor" which is essentially referring to Bayesian probability as belief.

Quoting from a post on vampires, "In Bayesian inference, you start
  with some initial beliefs (called 'Bayesian priors' or just 'priors'),
  and then you 'update' them as you receive new evidence."
This is the standard metaphor, and it's not so much wrong as
  unhelpful, misleading, and restricting. The metaphor derives from
  Bayes's rule (details which can be looked up anywhere) and which gives
  a formula which on the right-hand-side is supposed to be an element
  representing "prior beliefs." The formula itself is correct, as most
  math is. But because math is correct does not mean that it means what
  you think it means.

In my opinion, the prior being a "belief" is not a metaphor, but it is a veritable reflection of Bayesian probability. For instance, Bayesians can probabilistically quantify events that frequentists cannot such as the location of a quantum particle. I understand the distinction between Bayes' Rule and Bayesian Statistics: the former is a probabilistic law that applies to frequentist as well as Bayesian inference as well; the latter deals with a Bayesian interpretation of probability. He goes on to say:

This is wrong because there is no such thing as "Pr(Y)" or "Pr(X)".
  These objects do not exist. Numbers can be put in their place and the
  equation can be made to work out, but it is the step of putting
  numbers in that is wrong. There is no such thing as an unconditional
  probability, so we can never write without error "Pr(Y)" or "Pr(X)".
  Instead, we should write e.g. Pr(Y|W) or Pr(X|W), where W is the
  knowledge we start with, i.e. our real prior (knowledge).

And I think at this point--that this post doesn't seem to have anything to do with Bayesian statistics at all. Like a frequentist putting uncertainty bounds on uncertainty bounds, it seems the Bayesian analogue is conditioning on conditioning.
Is this blog sound reasoning? Is there actually a distinction between "belief" and "knowledge" when specifying a prior? If a conditional density exists, such as $Pr(Y|W)$ then a marginal density exists as well $Pr(Y) = \int Pr(Y|W)Pr(W)$?
Comment: It may be of interest to note the rhetorical subterfuge in the second quotation: if $(X,W)$ is not itself a random variable, then "$P(X\mid W)$" does not refer to any conditional probability: despite using the *notation* of conditional probability, it's merely a demand to make explicit the basis for our assessment of $P(X)$ by mentioning $W$ (and it's a blunder to refer to $W$ then as "our ... prior"). But when $(X,W)$ is itself random, this creates an infinite regress, because then we must invoke a third variable $W_1$ to discuss $P(W\mid W_1),$ and so on *ad infinitum.*
Comment: @whuber it's turtles all the way down.
Comment: Exactly what I was thinking :-)
Answer: Perhaps what the author was getting as is distinguishing between the posterior distribution and an individual's posterior distribution.
Suppose we knew as a fact that some of our parameters followed a given distribution before the data was distributed. Then based on those parameters, we observed some data conditional on those parameters and we knew the form of that conditional distribution. We could then apply Bayes theorem and know the probability distribution of said parameters, conditional on the data we saw.
On the other hand, traditional Bayesian statistics tells us that our uncertainty can be a prior distribution. That's fine and I think most modern statisticians have zero problem with that concept. However, it's sometimes swept under the rug that the final output is still conditional on the original prior, and generally speaking there's no reason to believe the next person will have the same prior as you.
To illustrate, consider the following R/psuedocode to demonstrate:
# Simulate mean first, then simulate data
simData = function(n = 10){
  # Simulate uniform(0,1)
  rand_unif = runif(1, min = 0, max = 1)
  # Mu is either -1 or 1 with probability 0.5
  if(rand_unif > 0.5){ mu = 1 }
  else{ mu = -1 }
  #Simulate
  output = rnorm(n, mu = mu, sd = 1)
  return(output)
}

Now if I showed you this code in advance and run the function, you can easily compute the probability that mu = 1 in the given simulation, despite never seeing it. This will unquestionable be correct (up to numerical error).
On the other hand, suppose you know the code up to
# Mu is some value <cliffsFavoriteNumber>
simData = function(n = 10){
  mu = cliffsFavoriteNumber
  #Simulate
  output = rnorm(n, mu = mu, sd = 1)
  return(output)
}

People other than me don't know the distribution on cliffsFavoriteNumber. Maybe Joe says "I don't think Cliff would choose a big number and it's gotta be a positive integer, so I'd say the prior for mu is a rounded exponential". Bob thinks "Cliff loves multiples of 9, so I'd say uniform prior on 9, 18, ..., 81". The computations they can compute from there are not mathematically wrong, but they  condition on different people's prior believes and as such end up with very different answers. Thus, it's a little misleading for Joe to say something like "The posterior distribution is ...". It's more precise to say something like "Conditional on believing that the prior distribution on mu is a rounded exponential, the posterior distribution will be ...".
Answer: The problem is that you W can't be integrated out. Taken all the knowledge you have, W, we form our prior, p(Y|W). And with this, we can use Bayes theorem to get our posterior p(Y|XW). W just tags along with all our updates, but we don't actually know p(W). If we did have the probability of our prior beliefs we could integrate W out, but that probably would itself be conditioned on its own prior beliefs. No matter what there are always prior beliefs that can't be integrated out.
I would like to note that many places don't include background knowledge as part of Bayes theorem, but it is there. Bayes theorem as properly expressed should read
$p(y|xw)=\frac{p(x|yw)p(y|w)}{p(x|w)}$
See how w just tags along?
Answer: If the prior accurately reflects your beliefs about the parameter and you are willing to make the assumption that the model for the data is correct, then the posterior is the rational way to update your beliefs after seeing the data.
The assumption that your model for the data is correct is implicit and sometimes not discussed, though it doesn't make sense to use conditional probability notation to make the assumption.  Admittedly though, this would draw more attention to the assumptions being made that are sometimes not discussed explicitly.
Similarly, $P(X)$ is understood to mean the marginal probability of the data averaged over your prior beliefs.
Saying $P(Y)$ does not exist is odd, though it is true that it can be difficult to construct $P(Y)$ in such a way that it does accurately reflect your prior beliefs.


Does the term Row-Complete have any synonyms?

Question: I'm wondering if there is other terminology that describes row-completeness, outside the context of a latin square, or if row-complete is actually a general term.
Answer: For those who are unfamiliar, a Latin square $L=(l_{ij})$ is an n by n matrix in which each symbol occurs exactly once in each row an each column, such as the following:
<code>0 1 5 2 4 3
1 2 0 3 5 4
5 0 4 1 3 2
2 3 1 4 0 5
4 5 3 0 2 1
3 4 2 5 1 0
</code>
A Latin square L is called row-complete if there are n(n-1) distinct tuples of the form $(l_{ij},l_{i(j+1)})$ in L.  The above Latin square is a row-complete Latin square (sourced from http://designtheory.org/library/encyc/exs/clsex.html) since if we look through it we find each ordered pair (0,1), (0,2), ..., (5,4) exactly once.  These Latin squares (and Latin squares in general) are sometimes used in statistical experiments (or so I'm led to believe).
For which orders n does there exist a row-complete Latin square is still not completely resolved (although some partial results are known).  In the abstract of J. T. Higham's PhD thesis Construction methods for row-complete Latin squares, row-complete Latin squares are also known as roman squares.  I haven't heard this term previously myself.
I don't know of any cases where "row-complete" has been used in other contexts, and searching through Google and MathSciNet didn't yield anything.  There are probably equivalent notions used in graph theory but under a different name.
If you really need an answer to the question, my feeling is you're going to need to ask a bunch of experts in the field.  E.g. you could present your work at a seminar or conference, and ask the audience if they know of an alternative name or usage of "row-complete".


Google VPN not supporting claimed ciphers

Question: Trying to make an IPSec IKEv2 connection with Google Cloud VPN services.
According to them, they support AES-CBC ciphers for Encyprtion at phase 1, but at the moment of negotiation with the on-premises VPN gateway only AES-GCM ciphers are offered.
So I get the famous "no proposal chosen" IKE SA error.
Do I need to tweak something on Google Cloud?
Comment: With "tweaking something on the Google Cloud" Is not possible to tweak this feature. Since you've followed the right steps and yet the VPN log is still showing no-proposal-chosen error, which indicates that the Google Cloud VPN and your on-premises VPN gateway were unable to agree on a set of ciphers. Then reporting this as a real issue not caused by a misconfiguration is indeed the next step so it gain more visibility.
Answer: I believe that the most probable cause of this error is a cipher mismatch. Is possible that you may have a proposal mismatch in IKE SA (phase 1) and 2.
You can also try troubleshooting this following this guide.
Specifically the following statement:

If the VPN logs show a no-proposal-chosen error, this indicates that Cloud VPN and your on-premises VPN gateway were unable to agree on a set of ciphers. For IKEv1, the set of ciphers must match exactly. For IKEv2, there must be at least one common cipher proposed by each gateway. Make sure your on-premises VPN gateway is configured using supported ciphers.

And also check the following as per the troubleshooting guide.

Verify that the on-premises IP configured on the Cloud VPN gateway is correct.
Check if the IKE versions configured on the VPN gateways match.
Verify that traffic is flowing between the two VPN gateways in both directions. In the VPN logs, check for reported incoming messages from the other VPN gateway.
Check that the IKE versions configured are the same on both sides of the tunnel.
Check that the shared secret is the same on both sides of the tunnel.
If your on-premises VPN gateway is behind one-to-one NAT, ensure that the NAT device has been properly configured to forward UDP traffic to your on-premises VPN gateway on ports 500 and 4500. Your on-premises gateway must be configured to identify itself using the public IP address of the NAT device. Refer to on-premises gateways behind NAT for details.

Also check that the lifetime in Phase 1 (IKE) is set to Google recommended value of 36,600 seconds (10 hours, 10 minutes), and the lifetime in Phase 2 set to 10,800 seconds (3 hours).
If the tunnel is not established after that, consider filing a public issue against the Cloud Platform/Networking using the Google issue-tracking tool. Include as many details as possible, including steps to reproduce for so that this issue can get a better visibility as well as more sampling.
Comment: I don't see any part of this answer which addresses the question.
Answer: It sounds like an issue at the GCP side.
Whenever you find that some GCP services work not as expected or against the behaviour described in the documentation you could file an issue report at the Google Public Issue Tracker or reach Google Cloud Support.
In addition, you can always check status of GCP services at Google Cloud Status Dashboard
Answer: This troubleshooting sounds vague, without more information about Peer VPN gateway device and its configuration.
So, the best approach here would be to grab the configuration and device to understand its compatibility configuration.
Maybe the support for IKE fragmentation was not enabled. Some third-party vendor devices, such as firewalls configured for stateful packet inspection, do not permit the passthrough of User Datagram Protocol (UDP) fragments in case they are part of a fragmentation attack 1.
If all fragments are not passed through, Internet Key Exchange (IKE) negotiation fails because the intended responder for the virtual private network (VPN) tunnel cannot reconstruct the IKE packet and proceed with establishment of the tunnel.
An example of this behaviour can be seen in the Cisco 2821 router:
<code>show crypto isakmp sa detail
[TIMESTAMP]: ISAKMP:(0):Support for IKE Fragmentation not enabled
</code>
A solution to this would be to enable IKE fragmentation.


Lotus Domino routing error

Question: When I send an email to any group it goes into a particular person inbox automatically.Please guide me.I am describing here like I have a group like name abc news and mail id is <firstname.lastname@example.com> and another person is <firstname.lastname@example.com>. So when I send email to group <firstname.lastname@example.com> it automatically moves into <firstname.lastname@example.com>. Kindly guide me.
Comment: Please add Server- Version and EXACT examples. I guess that SMTP_Left_Dot_Never_Domain=1 might help, but your description is very vague, so that I could be completely wrong... see this link: http://www-01.ibm.com/support/docview.wss?rs=463&context=SSKTMJ&context=SSPQ69&context=SSKTWP&context=SSRNU3&context=SSRNUY&q1=mail+not+delivered+domain&q2=user&uid=swg21102170&loc=en_US&cs=utf-8⟨=en
Answer: Inbound recipient lookups are controlled by the Address lookup setting on the Router/SMTP - Basics tab of the Configuration Settings document (global or server specific). This setting determines the criteria that the Router uses when attempting to match the SMTP address on an incoming message to an entry in the $Users view.
The settings for "Address lookup" are:

Fullname then Local Part - (default) The Router first searches the Domino Directory for a match for the full Internet address (email@example.com). If no match is found, it searches the directory again, looking for a match for the local part of the address only.
Fullname only - The Router searches the Domino Directory for full Internet addresses only. For example, it searches for "email@example.com" but not for "user." If an exact match is not found and the domain suffix is equivalent to an Internet domain alias defined in the Global domain document, a secondary search is performed using the domain suffix of the primary Internet domain.
Local Part only - The Router searches the Domino Directory for a match of the local part of the Internet address, that is, the part before the @ symbol. Local part matching matches periods and underscores in the address with spaces in the directory.

See the help for more details, especially the "Exhaustive lookup" option, but you will probably set this to "Fullname only" (watch out for side effects if you have several internet domains in place).
If that does not fix your problem, please specify in more detail, how the wrongly routed emails are sent, received and routed. See Troubleshooting IBM Lotus Domino 8 mail routing issues for more tips on how you can debug this yourself.

Lotus Domino routing error


How do you find where a module's directory is located?

Question: I have the following:
I want to find where the module "commerce_physical" is located in the modules folder, but I can't seem to locate this module. I did a find and a grep to locate where its files might be, but that doesnt seem to work. What are some other ways to find the location of a drupal 7 module?
Comment: If you're using the Kickstart distribution it'll be in profiles/commerce_kickstart/modules/contrib
Answer: Look at the {system} table.  In particular, the name column will be the module name (ie, the basename of the .info and .module files).  The filename column will be the full filename, with path, to the .info file for the module, relative to the DOCROOT of the installation.  See system_schema() for more information.
Comment: thanks got it!! http://img32.imageshack.us/img32/5786/croppercapture224.jpg


Spectrum of randomly projected matrices

Question: Consider a random projection of some deterministic positive $m\times m$ Hermitian matrix $A$, defined as $B:=PUAU^\dagger P$, where the $m\times m$ unitary matrix $U$ is Haar random and $P$ is some fixed $m\times m$ projector with rank $n<m$. Does the spectrum of $B$ typically match approximately (up to normalisation) with the spectrum of $A$ ordered in decreasing order at the first $n$ eigenvalues?
(I'm not sure how to formulate the approximation precisely. Presumably I want the spectral distributions to be close as $n,m$ become large enough.)
Related post at mathoverflow, but it is not yet answered.
Comment: Could you say what you call Haar random vectors ?
Comment: You have written $B=PAP$. Isn't it $B=P^TAP$ ? Besides, could you give the resp. dimensions of $A,B,P$ ?
Comment: Do you mean "let $v_1,\cdots,v_k$ be the first $k$ columns of a Haar unitary, let $P$ be the orthogonal projection on the subspace they span, and let $A$ be a random Hermitian matrix (according to which law, btw?); can you help me understand the spectrum of $PAP$?"?
Comment: @JeanMarie A "projection" in this case refers to a matrix $P$ with $P = P^T = P^2$; in general, its codomain is distinct from its image. If the columns of $U$ form an orthonormal basis for the image of $P$, then $P = UU^T$ and $PAP$ will have the same non-zero eigenvalues as $U^TAU$.
Comment: @Ben Grossmann Thanks: it is much clearer. I am surprized by the discrepancy between this question and the (similar ?) question referenced in MathOverflow.
Comment: Sorry guys~ Hopefully now it is clearer and I think it is the same problem as in the MO post.
Comment: You haven't answered me about the definition of "Haar random vectors" which is not a classical term.
Comment: @JeanMarie I've reformulated the question that doesn't use "Haar random vectors" any more.
Comment: You say in this version that your matrix is positive (definite positive ?) but in the MO post, you only say that it is symmetric...
Comment: @JeanMarie well, that MO post isn't by me, consider what I asked here a slightly different version than the MO one.
Answer: The answer is no. The largest eigenvalue of $PAP$ will match that of $A$ if and only if the eigenvector $x$ of $A$ associated with the largest eigenvalue of $A$ is an element of the range of $P$. However, for a given vector $x$, a "random" projection $P$ will contain $x$ with probability zero since the set of such projections is a proper Zariski closed subset of the set of projections.

Spectrum of randomly projected matrices


Couldn't match type 'time-188.8.131.52:Data.Time.Clock.Internal.UTCTime.UTCTime' with 'UTCTime'

Question: I don't understand why I am getting this error using the thyme library:
λ import qualified Data.Thyme.Format as ABC (formatTime)
λ import qualified Data.Thyme.Clock as XYZ
λ import Data.Thyme.Time.Core (fromThyme)
λ let test123 :: XYZ.UTCView -> String; test123 p = ABC.formatTime defaultTimeLocale "%Y-%m-%d" (fromThyme p :: XYZ.UTCTime)

<interactive>:4:96: error:
    • Couldn't match type 'time-22.214.171.124:Data.Time.Clock.Internal.UTCTime.UTCTime'
                     with 'XYZ.UTCTime'
      NB: 'XYZ.UTCTime'
            is defined in 'Data.Thyme.Clock.Internal'
                in package 'thyme-0.3.5.5'
          'time-18.104.22.168:Data.Time.Clock.Internal.UTCTime.UTCTime'
            is defined in 'Data.Time.Clock.Internal.UTCTime'
                in package 'time-184.108.40.206'
        arising from a functional dependency between:
          constraint 'Data.Thyme.Time.Core.Thyme XYZ.UTCTime XYZ.UTCView'
            arising from a use of 'fromThyme'
          instance 'Data.Thyme.Time.Core.Thyme
                      time-184.108.40.206:Data.Time.Clock.Internal.UTCTime.UTCTime XYZ.UTCView'
            at <no location info>
    • In the third argument of 'formatTime', namely
        '(fromThyme p :: XYZ.UTCTime)'
      In the expression:
        formatTime
          defaultTimeLocale "%Y-%m-%d" (fromThyme p :: XYZ.UTCTime)
      In an equation for 'test123':
          test123 p
            = formatTime
                defaultTimeLocale "%Y-%m-%d" (fromThyme p :: XYZ.UTCTime)

Why is it referencing the UTCTime defined within the time library?
http://hackage.haskell.org/package/thyme-0.3.5.5/docs/Data-Thyme-Format.html#t:FormatTime
Answer: Because:
fromThyme :: Thyme b a => a -> b
instance Thyme Data.Time.Clock.UTCTime UTCView

You should probably use utcTime instead.

Couldn't match type ‘time-1.8.0.2:Data.Time.Clock.Internal.UTCTime.UTCTime’ with ‘UTCTime’


Is it possible to read only the first line of a file with Actionscript?

Question: I've got a PHP-based site where we allow users to upload a comma-separated text file of data to be imported by some server side scripts. However, sometimes we need to adjust the column order of the data that is uploaded, and it would be immensely helpful if we could identify the columns in the CSV file before the upload takes place.
The only way I can think of to access local file I/O is with a Java plugin or an embedded Flash script, and I'd much prefer the latter. Is there any way in Flash to select a text file on the local hard drive, and then use ActionScript to examine only the first line (which corresponds to the column headings, in this example)?
Sometimes these files can be >100 MB, so loading the whole thing into memory and then throwing out all but the first line probably wouldn't work efficiently, so I'm wondering if there's syntax in ActionScript to limit the amount of data read in at a time.
Answer: Flash movies don't have access to the local file system, except to do a file upload to the server.
I think your options are:

Process the class on the client side with Java (trivial), using an applet. However, as you rightly assert the user would need to grant permissions to the applet.
Provide a small client side application written with Adobe Air, using the file system API.
Answer: Flash Player 10, the current shipping version (just released 2 days ago), allows you to access local files. You can find an example here:
http://www.mikechambers.com/blog/2008/08/20/reading-and-writing-local-files-in-flash-player-10/
Hoewver, you have to load the entire file.
If you use a SWF running in Adobe AIR, then you can read just the first line (it has more advanced file APIs).
Best solution is probably to create a PHP API that returns just the first row of data.
mike chambers
email@example.com
Comment: I don't want to involve PHP with this at all. I will be uploading the file to PHP either way, but I wanted something client-side to process just the first row of data before the actual upload takes place, because sometimes the files can be quite large.
Answer: The current version of the Flash player is only able to select a file for upload to a server, requiring all processing to be server-side. The next version of the player however, Flash 10 can read local files without making the round trip to a server.
I haven't used this functionality myself (yet) so I'm not sure if it's possible to read just a part of the file, but that does become less of an issue if it's all local.
Flash 10 is currently in some sort of prerelease beta, but it will roll out soon I think.
EDIT: Soon indeed, just saw this in my feed reader


What is the win32 equivalent of Process.MainModule.FileName

Question: Trying to find a win32 process location and not just the name.
Can't seem to see any literature online about this?
Current Script is:
<code>                Process[] localByName = Process.GetProcessesByName("lmgrd_x64_n6");
        foreach (Process proc in localByName)
        {
            try
            {
                try
                { //64bit
                    lb_instances.Items.Add(proc.MainModule.FileName + " " + proc.Id);
                }
                catch (Exception ex)
                { //32bit
                    lb_instances.Items.Add(proc.ProcessName + " " + proc.Id);
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.ToString(), ex.InnerException.ToString());
            }
</code>
I get exception error:

A 32 bit processes cannot access modules of a 64 bit process.
Comment: https://msdn.microsoft.com/en-us/library/windows/desktop/ms683217(v=vs.85).aspx
Comment: The following will return the paths of the first 100 processes, no matter how they were created. It returns 32 and 64 bit processes, system services like `svchost.exe`, user applications like Chrome etc. Just make sure you run it with *elevated* priviledges:  `System.Diagnostics.Process.GetProcesses().Take(100).Select(p=>{try{return p.MainModule.FileName;}catch{return "";}})`. I used LinqPad to run this application
Comment: That's a *different* issue. It doesn't mean that `MainModule` doesn't work for 32-bit applications. It means that you can't get that info for *64-bit* applications if you are running in 32-bit. For example, LinqPad is a 64-bit application, which is why it was able to retrieve info for all kinds of processes. You can simply change your target to x64 or clear the `Prefer 32-bit` checkbox in your build properties.
Comment: Perhaps you should explain what you mean by `Win32`, `process location` and `Process.MainModule.FileName`? What script are you referring to? Win32 is the API, not a different kind of process. All executables behave the same, no matter the language they were written on
Comment: Are you referring to [Process.MainModule](https://msdn.microsoft.com/en-us/library/system.diagnostics.process.mainmodule(v=vs.110).aspx)? That should return the executable for 32 and 64 bit applications
Comment: @PanagiotisKanavos I get the error ' A 32 bit processes cannot access modules of a 64 bit process.'
Comment: @SCramphorn how did you get hold of the process object in the first place?
Comment: @PanagiotisKanavos I've added my code just before then if that helps!
Comment: Anoter option is to use WMI as shown in the duplicate question
Comment: @PanagiotisKanavos thanks, that fixed my issue - please post as an answer and I will accept it!

Regards,
SAm
Answer: You can call Module32First and grab the path from the structure. That's how .NET does it, anyway.


AADSTS90019 when attempting automatic Azure AD registration of domain-joined Windows 10 device

Question: I am attempting to set up automatic AAD join for Windows 10 as described here: https://azure.microsoft.com/en-gb/documentation/articles/active-directory-conditional-access-automatic-device-registration-setup/
We have two internal ADFS 3.0 servers (Server 2012R2). They are configured using Azure AD Connect for federation with Office 365 on four UPNs:

ad.dom1.com - this is the forest name, we only have a single forest
dom1.com - most users exist under this domain
dom2.com
dom3.com

The ADFS servers are exposed using a TCP-level load balancer on https://adfs.ad.dom1.dom, with a certificate signed by a public CA. The ADFS servers are not running the DRS, as we are intending to do that with Azure AD.
The federated authentication with Office 365 is successful for users created with any of those UPN suffixes, but only after having altered the third rule as described in https://blogs.technet.microsoft.com/abizerh/2013/02/05/supportmultipledomain-switch-when-managing-sso-to-office-365/
All of the prerequisite steps on the Azure article have been performed:

Set the service connection point
Executed Initialize-ADSyncDomainJoinedComputerSync
Ensured that first three federation rules in the article exist (they were created automatically by Azure AD Connect)
Ensured that Auth Method Claim Rule exists and executed Set-AdfsRelyingPartyTrust
Created the Group Policy

Additionally, the domains:

enterpriseregistration.dom1.com
enterpriseregistration.ad.dom1.com
enterpriseregistration.dom2.com
enterpriseregistration.dom3.com

Are all CNAMEs for enterpriseregistration.windows.net
However, while all other authentication seems to work fine, the automatic AADJ process fails on all existing Windows 10 Enterprise domain joined client machines. The following errors are present in the Microsoft/Windows/User Device Registration event log:
Event ID 305
Automatic registration failed at authentication phase.  Unable to acquire access token.  Exit code: Unspecified error. Server error: AdalMessage: GetStatus returned failure
AdalError: invalid_request
AdalErrorDesc: AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z
AdalErrorCode: 0xcaa90006
AdalCorrelationId: <uuid>
AdalLog:  HRESULT: 0xcaa90006
AdalLog:  HRESULT: 0xcaa20002
AdalLog: Webrequest returns error code:invalid_request and error description:AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z ; HRESULT: 0x0
AdalLog: Token response is not successfull. Status:400 ResponseText:{"error":"invalid_request","error_description":"AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.\r\nTrace ID: <uuid>\r\nCorrelation ID: <uuid>\r\nTimestamp: 2016-11-14 12:30:28Z","error_codes":[90019],"timestamp":"2016-11-14 12:30:28Z","trace_id":"<uuid>","correlation_id":<uuid>"} ; HRESULT: 0x0
AdalLog: WebRequest Status:400 ; HRESULT: 0x0
AdalLog: Webrequest has valid state ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest opening connection ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth- received realm info ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth w Tenant ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- returns false ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- refresh token is not available ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken get refresh token info ; HRESULT: 0x0
AdalLog: Authority validation is completed ; HRESULT: 0x0
AdalLog: Authority validation is enabled ; HRESULT: 0x0
AdalLog: Token is not available in the cache ; HRESULT: 0x0
. Tenant Type: dom1.com

Event ID 304
Automatic registration failed at join phase.  Exit code: Unknown HResult Error code: 0xcaa1000e. Server error: empty. Debug Output:\r\n joinMode: Join
drsInstance: azure
registrationType: fed
tenantType: fed
tenantId: <uuid>
configLocation: undefined
errorPhase: auth
adalCorrelationId: <uuid>
adalLog: AdalLog:  HRESULT: 0xcaa1000e
AdalLog:  HRESULT: 0xcaa90006
AdalLog:  HRESULT: 0xcaa20002
AdalLog: Webrequest returns error code:invalid_request and error description:AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z ; HRESULT: 0x0
AdalLog: Token response is not successfull. Status:400 ResponseText:{"error":"invalid_request","error_description":"AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.\r\nTrace ID: <uuid>\r\nCorrelation ID: <uuid>\r\nTimestamp: 2016-11-14 12:30:28Z","error_codes":[90019],"timestamp":"2016-11-14 12:30:28Z","trace_id":"<uuid>","correlation_id":"<uuid>"} ; HRESULT: 0x0
AdalLog: WebRequest Status:400 ; HRESULT: 0x0
AdalLog: Webrequest has valid state ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest opening connection ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth- received realm info ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth w Tenant ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- returns false ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- refresh token is not available ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken get refresh token info ; HRESULT: 0x0
AdalLog: Authority validation is completed ; HRESULT: 0x0
AdalLog: Authority validation is enabled ; HRESULT: 0x0
AdalLog: Token is not available in the cache ; HRESULT: 0x0

adalLog: AdalLog:  HRESULT: 0xcaa1000e
AdalLog:  HRESULT: 0xcaa90006
AdalLog:  HRESULT: 0xcaa20002
AdalLog: Webrequest returns error code:invalid_request and error description:AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z ; HRESULT: 0x0
AdalLog: Token response is not successfull. Status:400 ResponseText:{"error":"invalid_request","error_description":"AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.\r\nTrace ID: <uuid>\r\nCorrelation ID: <uuid>\r\nTimestamp: 2016-11-14 12:30:28Z","error_codes":[90019],"timestamp":"2016-11-14 12:30:28Z","trace_id":"<uuid>","correlation_id":"<uuid>"} ; HRESULT: 0x0
AdalLog: WebRequest Status:400 ; HRESULT: 0x0
AdalLog: Webrequest has valid state ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest opening connection ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth- received realm info ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth w Tenant ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- returns false ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- refresh token is not available ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken get refresh token info ; HRESULT: 0x0
AdalLog: Authority validation is completed ; HRESULT: 0x0
AdalLog: Authority validation is enabled ; HRESULT: 0x0
AdalLog: Token is not available in the cache ; HRESULT: 0x0

adalResponseCode: 0xcaa1000e
.

dsregcmd.exe
Similar errors appear if I try to run C:\windows\system32\dsregcmd.exe /debug from a SYSTEM command prompt:
dsregcmd::wmain logging initialized.DsrCmdAccountMgr::IsDomainControllerAvailable DsGetDcName success { domain:ad.dom1.com forest:ad.dom1.com domainController:\\ldndc01.ad.dom1.com isDcAvailable:true }
PreJoinChecks Complete.
preCheckResult: Join
isPrivateKeyFound: undefined
isJoined: undefined
isDcAvailable: YES
isSystem: YES
keyProvider: undefined
keyContainer: undefined
dsrInstance: undefined
elapsedSeconds: 1
resultCode: 0x0
Automatic device join pre-check tasks completed.TenantInfo::Discover: tenant type detection, validating https://adfs.ad.dom1.com/adfs/ls/
TenantInfo::Discover: tenant type detection, checking match against https://login.microsoftonline.com
TenantInfo::Discover: tenant type detection, checking match against https://login.windows-ppe.net
TenantInfo::Discover: Join Info TenantType:Federated  AutoJoinEnabled:1 TenandID:<uuid> TenantName:dom1.com

DsrCmdSettings::GetSetting: The key was not found, so returning FALSE. Key: SOFTWARE\Microsoft\Windows\CurrentVersion\CDJ
AdalLog: Token is not available in the cache ; HRESULT: 0x0
AdalLog: Authority validation is enabled ; HRESULT: 0x0
AdalLog: Authority validation is completed ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken get refresh token info ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- refresh token is not available ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- returns false ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth w Tenant ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth- received realm info ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest opening connection ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest has valid state ; HRESULT: 0x0
AdalLog: WebRequest Status:400 ; HRESULT: 0x0
AdalLog: Token response is not successfull. Status:400 ResponseText:{"error":"invalid_request","error_description":"AADSTS90019: No tenant-identifying information found in either the request or implied by any provided
credentials.\r\nTrace ID: <uuid>\r\nCorrelation ID: <uuid>\r\nTimestamp: 2016-11-14 12:30:28Z","error_codes":[90019],"timestamp":"2016-11-14 12:30:28Z","trace
_id":"<uuid>","correlation_id":"<uuid>"} ; HRESULT: 0x0
AdalLog: Webrequest returns error code:invalid_request and error description:AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z ; HRESULT: 0x0
AdalLog:  HRESULT: 0xcaa20002
AdalLog:  HRESULT: 0xcaa90006
AdalMessage: GetStatus returned failure
AdalError: invalid_request
AdalErrorDesc: AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z
AdalErrorCode: 0xcaa90006
AdalCorrelationId: {39AEBF80-8679-4A5A-86D3-409CB1A8D8EF}
AdalLog:  HRESULT: 0xcaa90006
AdalLog:  HRESULT: 0xcaa20002
AdalLog: Webrequest returns error code:invalid_request and error description:AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: <uuid>
Correlation ID: <uuid>
Timestamp: 2016-11-14 12:30:28Z ; HRESULT: 0x0
AdalLog: Token response is not successfull. Status:400 ResponseText:{"error":"invalid_request","error_description":"AADSTS90019: No tenant-identifying information found in either the request or implied by any provided
credentials.\r\nTrace ID: <uuid>\r\nCorrelation ID: <uuid>\r\nTimestamp: 2016-11-14 12:30:28Z","error_codes":[90019],"timestamp":"2016-11-14 12:30:28Z","trace
_id":"<uuid>","correlation_id":"<uuid>"} ; HRESULT: 0x0
AdalLog: WebRequest Status:400 ; HRESULT: 0x0
AdalLog: Webrequest has valid state ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: Webrequest opening connection ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth- received realm info ; HRESULT: 0x0
AdalLog:  HRESULT: 0x4aa90010
AdalLog: AggregatedTokenRequest::UseWindowsIntegratedAuth w Tenant ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- returns false ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken- refresh token is not available ; HRESULT: 0x0
AdalLog: AggregatedTokenRequest::AcquireToken get refresh token info ; HRESULT: 0x0
AdalLog: Authority validation is completed ; HRESULT: 0x0
AdalLog: Authority validation is enabled ; HRESULT: 0x0
AdalLog: Token is not available in the cache ; HRESULT: 0x0
AdalLog:  HRESULT: 0xcaa1000e
wmain: Unable to retrieve access token 0x80004005.
DSREGCMD_END_STATUS
        AzureAdJoined : NO
     EnterpriseJoined : NO

Comment: Don't sanitize correlation id guids. They are necessary to view backend logs.
Answer: I understand your frustration with this. I just spent about 20 hours troubleshooting problems with automatic AAD joining. I am running the latest version of AD Connect, and am running an ADFS farm on Server 2016. I did NOT let AD Connect configure my ADFS servers.
I had the exact same error. The issue was a missing ImmutableID claim. This link proved to be the best resource for setting up azure AD join: https://docs.microsoft.com/en-gb/azure/active-directory/active-directory-conditional-access-automatic-device-registration-setup
There is a script listed there to automatically add the necessary claim rules. Now that i practically understand this whole process inside and out, I can say that the script actually does work and adds the correct claims rules.
However, what is misleading is how to configure the couple of variables in the script. So, I thought I would clarify based on my learning, hopefully it will save somebody some time.

$MultipleVerifiedDomainNames: The description and name is both WRONG
and MISLEADING. Set this to $TRUE only if you have MULTIPLE FEDERATED
DOMAINS in your Office 365 tenant.
$immutableIDAlreadyIssuedforUsers:
If the Immutable ID (Source Anchor) for your AD Connect
synchronization does NOT use objectGUID then set this to $TRUE.
$oneOfVerifiedDomainNames: If you set $MultipleVerifiedDomainNames to
$true, then set this to the Office 365 verified domain name that you
want your devices to register to.

Do not change any other components of the script, and make sure you only run it once. If you need to run it again, you need to manually remove the added claims issuance rules from the RP trust or they will be duplicated.
Another very helpful thing to know when it comes to troubleshooting on Windows 10, use DSREGCMD. It has to run as SYSTEM so you'll need something like PSEXEC.
psexec -i -s cmd.exe
dsregcmd /debug
This will force an immediate registration to Azure, and report detailed information about the failure.
During my testing, Windows 7 worked fine, but Windows 10 would not AD join. If the ImmutableID is the problem you'll see the error is: AADSTS90019: No tenant-identifying information found in either the request or implied by any provided credentials.
If, you included a verified domain when you shouldn't have or it's wrong you'll see: AADSTS50107: Requested federation realm object your specified domain does not exist.
Answer: Firstly, please note that this process is called Automatic AAD registration or Automatic workplace join, not Automatic AAD join. AAD Join are different with AAD registration, that's a feature only for Win10 (professional or enterprise editions).
I have test this in my lab and successfully completed the automatic registration for my Server2012 R2 and Win10 machines to AAD via the MSI package & GPO. In the first scenario(via MSI package), the schedule task will be triggered when login through the user account which has been synced to AAD via AAD connect. After the task completed, you will find the devices have been registered to AAD and associated with that user.

In the second scenario via GPO, the event log showed that the Automatic Registration is completed and also I can see the machines in Azure AD portal.

For your issue, I think you can try to re-add claim rules($rule1~$rule3) on your ADFS server ahthough they are already existed. Also please ensure that the ADFS server in your environment is configured and worked correctly.


Simple condition for asymptotic normality of MLE

Question: Are these conditions sufficient for asymptotic normality of MLE? If it is, pls. let me know the references.
1-First and second derivatives of $\ell(\theta,\eta)$ are defined.
2- The Fisher information matrix be non-singular and continuous with respect to the parameters $\theta$ and $\eta$.
Comment: Another condidion is that your parameter cannot be on the boundary of its domain, so you might want to revise to have the first and second derivatives of $\ell(\theta,\eta)|_{\theta^*,\eta^*}$ be the goal. Also, a simple test is to see if the log-likelihood function approaches a quadratic as n $\rightarrow \infty$
Comment: Why don't you try proving the asymptotic normality of the MLE using this set of assumptions and see if you need any more?
Comment: Dear Eupraxis 1981 ... I just want know if these 2 conditions are sufficient for asymp. Normality or not
Comment: Dear fg nu ... which set of assumption?
Answer: Asymptotic normality is usually proven for a local maximum of the likelihood function. I paste below the conditions as stated in T. Amemiya (1985) Advanced Econometrics, ch. 4, for extremum or $M$-estimators in general. $T$ is sample size, $\theta$ is the unknown parameter vector, and  $\theta_0$ is the true values, $\mathbf y$ is the data sample, $Q_T(\mathbf y, \theta)$ is the objective function, $\hat \theta_T$ is the ML estimator. $\Theta_T$ is the set containing the roots of the equation $\frac {\partial Q_T(\mathbf y, \theta)}{\partial \theta} =0$ corresponding to local maxima.

Variations do exist. For example, if $Q_T(\mathbf y, \theta)$ is concave over the parameter space for any $\mathbf y$, then uniform convergence (4.1.2.-C) can be replaced by point-wise convergence. Also, there are alternative ways to express some of the conditions. For example condition 4.1.3-B may be replaced by its own sufficient condition, the local dominance condition : for some neighborhood of $\theta_0$, $N(\theta_0)$,
$$E\left[\text {sup}_{\theta \in N(\theta_0)}  ||\partial^2Q_T/\partial \theta\partial\theta'||\right] <\infty$$.
ADDENDUM
Responding to @whuber's comment (my meta-response can be found in the comments), consider an i.i.d sample $\{x_i,\, i=1,...,n\}$ from a uniform $U(0,\theta)$. Then the ML estimator for $\theta$ is
$$\hat \theta_{ML} = \max_i x_i = X_{(n)}$$
The asymptotic distribution of the maximum of an i.i.d sample is known not to be normal. This is an example that asymptotic normality requires more conditions than those stated in the question.
Comment: @whuber Another logically valid answer is to use a reliable source that states the full set of conditions, which is what I did. Then, if these conditions are more numerous than the conditions stated in the question, it follows that the latter are not sufficient. Moreover my answer, by providing the full set of conditions, could help future users that may wonder "what are the sufficient conditions for asymptotic normality to hold", and may come to CV looking for an answer. Certainly, adding counter examples would make for a fuller and more educational answer... so I added one such example.
Comment: That this is not a logically valid approach is exemplified by the Central Limit Theorem, for which numerous non-equivalent sets of conditions are known. "More numerous" is not meaningful, either (although "logically is implied by" would be). I do not see anything in the material you quoted that claims these are necessary *and sufficient* conditions, which is what you seem to be talking about here. Moreover, your addendum is not a counterexample because its log likelihood fails to be differentiable everywhere.
Comment: The question posits two conditions and asks whether they are sufficient. Thus, a logically valid answer will either (a) demonstrate they are sufficient or else (b) provide a counterexample in which the two given conditions hold but asymptotic normality does not.


Pass variable to Python Flask form

Question: I'm currently stuck trying to build a variable message flashing function for my Flask forms
Since I'm building a tool that has to have multiple languages available, I wish to create custom messages for my flask form validation based on specific language input.
My form looks something like this:
class messageForm(FlaskForm):
    title = StringField(
        'title',
        validators=[
            validators.DataRequired(validationMessage('validateRequired', language=language))
        ]
    )

the function "validationMessage" looks like this:
def validationMessage(message, language):
    msg = Message.query.filter_by(title=message).first()
    lang = Language.query.filter_by(abbr=language).first()
    text = messageBody.query.filter_by(message_id=msg.id, language_id=lang.id).first().text
    return text

As you notice, I do some lookup in a few database tables to produce my message text.
My trouble is now...
How do I pass my language variable from my view to my form so that I can pass it on to the "validationMessage" function?
The language variable is based on a variable in my view endpoint
# Messages
@admin.route('/<string:language>/message', methods=['GET'])
def messageView(language='dk')
    form=messageForm()
    ...

I have considered using my session for this, but as I understand it, I can only use this within my view and therefore not within either my form or the message function
Comment: I'm assuming you're using flask-wtforms? If so, the form can be instantiated with kwargs for the fields. So if you had a `HiddenField` called `lang`, your form could be instantiated as `form=messageForm(lang=language)`. Then set the validators `language` argument to be the value of your `HiddenField`
Comment: You could always add a HiddenField to the form and then set that parameter after you construct it in your view. The value should be then available to you in your form instance
Comment: I could do this, if I needed to pass the variable through to my template, unfortunately I need to pass the variable to my Form class, and as such it won't work for me.
Comment: Trying this I unfourtunately get the following error:
Comment: The problem is actually that the `validationMessage` function only gets called when initializing your validator. And that happens right when you start your Flask app. It is not called later on form validation anymore. So this approach won't work at all.
Answer: You can solve this problem by using the global context variable g. The trick is not to use the builtin validator but a custom validator. The reason is that the builtin validator is a factory function. This means the validationMessage function you pass to it will only be executed on class creation. If you build a custom validator and read the language from the global context variable it will work.
def custom_validator(form, field):
    language = g.language
    msg = Message.query.filter_by(title=message).first()
    lang = Language.query.filter_by(abbr=language).first()
    text = messageBody.query.filter_by(message_id=msg.id, language_id=lang.id).first().text

    if not len(field.data):
        raise validators.ValidationError(text)

Replace the validator in your form with the custom validator:
class messageForm(FlaskForm):
    title = StringField('title', validators=[custom_validator])

In the view function just create the language property for the global context variable.
# Messages
@admin.route('/<string:language>/message', methods=['GET'])
def messageView(language='dk')
    g.language = language
    form=messageForm()
    ...

Comment: Does this also work for custom `choices` argument passed to a `SelectField`? If yes, how to create a custom choices function? I tried it with a function using global variables returning the choices where the flobal variable is set at the beginning of the main() function. But this does not work

Pass variable to Python Flask form


Reference request: infinite-dimensional manifolds

Question: The following books and/or notes develop various aspects of the theory of infinite-dimensional manifolds:

Lang, Fundamentals of Differential Geometry.
Kriegl & Michor, The Convenient Setting of Global Analysis.
Choquet-Bruhat & DeWitt-Morette, Analysis, Manifolds and Physics.
Klingenberg, Riemannian Geometry.
Marsden, Ratiu, and Abraham, Manifolds, Tensor Analysis, and Applications.
Hamilton, The inverse function theorem of Nash and Moser.

Question: Are there any other books that systematically develop from scratch the theory of infinite-dimensional manifolds, in particular Frechet manifolds?
Answer: It's not specifically a text on infinite-dimensional manifolds, but the theory of diffeology developed in Patrick Iglesias-Zemmour's text encompasses infinite dimensional manifolds of all sorts (Banach, Frechet, etc.). The category of diffeological spaces is also a quasi-topos, which is a really fantastic property.
Answer: Infinite-Dimensional Lie Groups by Karl-Hermann Neeb looks really nice. It includes basics of the theory of infinite-dimensional manifolds in chapter 2. He also shows how the theory of linear ODEs collapses for Frechet spaces.


Why are these definitions of directional derivative equivalent? (By limit and by scalar product)

Question:
I want to prove that the definitions of directional derivative by scalar product, $\nabla f(x) .v$, and by a limit, $\displaystyle \lim_{h \to 0}\dfrac{f(x+hv)-f(x)}{h}$, are the same one. I tried to do it using definitions of gradient but failed. Wikipedia proved it this way, but I can't understand the start of the proof, because I don't see why you can start from that limit being $0$. Isn't this an assumption of the thesis?
Note: While I was writing this post, I found an answer on why they are equivalent here, but I still want an explanation of what Wikipedia is doing, so I'll make the post.
Answer: This topic is a minefield when it comes to notation inconsistencies if you're not careful.
The reason why Wikipedia set the answer of the limit of the last equation to 0 is because it uses the definition of Fréchet differentiability, which you can verify by clicking the blue differentiable link in your image.
In the implication part, the first equality is due to the fact that the gradient of a scalar function creates an R^n vector at any given x, which is a linear map satisfying the definition of Fréchet differentiability. The second equality comes straight from the algebra from above.
Comment: Fréchet differentiability implies directional derivative.
Comment: If it is Fréchet differentiability, then why is it $t$ and not $||h||=||tv||=|t|||v||=|t|$ in the denominator?


HSB smooth color transition

Question: I made a Happy Mothers day program that involved text slowly changing colors. I'm using HSB, and slowly changed the Hue value until I got to 255 then jumped back to 0. But this doesn't seem to give that smooth color transition I'm looking for.
This is basically what I'm doing:
fill(clamp(frameCount*0.2, 255), 255, 255);

function clamp(c, cap){

  do { c -= cap } while ( c > cap );

  return c

}

Full program: https://editor.p5js.org/KoderM/sketches/RekPOFctj
Does anybody know how do get a smooth rainbow effect with HSB color values?
Answer: One of the problems with the code wasn't shown. It's the colorMode call, which was set to the default colorMode(HSB), which, as the docs say:

By default, this is colorMode(HSB, 360, 100, 100, 1)

function setup() {
  createCanvas(windowWidth, windowHeight);
  colorMode(HSB);
}

function draw() {
  fill(frameCount % 360, 100, 100);
  rect(0, 0, windowWidth, windowHeight);
}

You can also use colorMode(HSB, 255); and override the default, then use fill(frameCount % 255, 255, 255); (with optional scaling if desired).
Also, clamping isn't really the correct concept. That's traditionally a min/max algorithm, but we actually want to wrap around to 0.

HSB smooth color transition


Why are there two player speeds in my game?

Question: For some reason there are two player speeds in my game Orange Quest. What I mean by this is when I test my game in eclipse and my headphones are off. The game goes twice as fast as when my headphones are on.
I sent the game to my friend and he also experiences the same problems. I've tried to debug the code to see if there was a something wrong with it, but I didn't see any problem. I also tried to adjust the player speed but that didn't help either, as it was still going twice as fast when my headphones were off.
Here is the code for the game panel
Oh yah, by the way the frame is 700 by 700
import javax.imageio.ImageIO;

import javax.swing.*;
import java.awt.event.*;
import java.awt.image.BufferedImage;
import java.awt.*;
import java.util.*;
import java.util.concurrent.CopyOnWriteArrayList;

public class GamePanel extends JPanel implements ActionListener {
    /**
     *
     */
    private static final long serialVersionUID = 1L;

    Player player;

    boolean running = true;
    CopyOnWriteArrayList<Wall> walls = new CopyOnWriteArrayList<>();
    CopyOnWriteArrayList<LavaWall> lavaWalls = new CopyOnWriteArrayList<>();
    CopyOnWriteArrayList<Enemy> enemies = new CopyOnWriteArrayList<>();
    CopyOnWriteArrayList<Orange> oranges = new CopyOnWriteArrayList<>();

    java.util.Timer gameTimer;
    int cameraX;
    int offset;
    int count = 0;
    int index = 0;
    int set;
    int s = 50;
    int r = 75;
    BufferedImage dirt;
    BufferedImage grass;
    BufferedImage lava;
    BufferedImage cloud;
    BufferedImage spike;
    BufferedImage undergroundLava;
    BufferedImage character;
    BufferedImage thumbnail;
    JLabel timeLabel = new JLabel();
    JLabel orangeLabel = new JLabel(": 0/7");
    double orangeCount = 0;
    int elapsedTime = 0;
    int minutes = 0;
    int seconds = 0;
    int ms = 0;
    String minutesString = String.format("%02d", minutes);
    String secondsString = String.format("%02d", seconds);
    String msString = String.format("%02d", ms);
    java.util.Timer timer = new java.util.Timer();
    TimerTask task = new TimerTask() {

        @Override
        public void run() {
            elapsedTime += 10;
            minutes = (elapsedTime / 60000);
            seconds = (elapsedTime / 1000) % 60;
            ms = (elapsedTime / 10) % 100;
            minutesString = String.format("%02d", minutes);
            secondsString = String.format("%02d", seconds);
            msString = String.format("%02d", ms);
            timeLabel.setText(minutesString + ":" + secondsString + ":" + msString);
        }

    };

    GamePanel() {

        if (running) {
            player = new Player(400, 300, this);
            reset();
            setLayout(null);
            gameTimer = new java.util.Timer();
            add(player.death);
            player.death.setBounds(550, 15, 300, 100);
            player.death.setFont(new Font("Verdana", Font.PLAIN, 20));
            add(orangeLabel);
            orangeLabel.setBounds(320, 15, 200, 50);
            orangeLabel.setFont(new Font("Verdana", Font.PLAIN, 20));
            gameTimer.schedule(new TimerTask() {

                @Override
                public void run() {

                    try {
                        if (walls.get(walls.size() - 1).x < 800) {
                            offset += 700;
                            makeWalls(offset);
                        }

                        player.set();
                        for (Wall wall : walls)
                            wall.set(cameraX);
                        for (int i = 0; i < walls.size(); i++) {
                            if (walls.get(i).x < -800)
                                walls.remove(i);
                        }for(Enemy enemy : enemies)
                            enemy.set(cameraX);
                        for (LavaWall lava : lavaWalls)
                            lava.set(cameraX);
                        for (int i = 0; i < lavaWalls.size(); i++) {
                            if (lavaWalls.get(i).x < -800)
                                lavaWalls.remove(i);
                        }
                        for(Orange orange : oranges)
                            orange.set(cameraX);
                        for(int i = 0; i < oranges.size(); i++) {
                            if(oranges.get(i).x < -800)
                                oranges.remove(i);
                        }
                        repaint();

                    } catch (Throwable e) {
                        e.printStackTrace();
                    }
                }
            }, 0, 17);
            add(timeLabel);
            timeLabel.setText(minutesString + ":" + secondsString + ":" + msString);
            timeLabel.setBounds(10, 15, 300, 100);
            timeLabel.setFont(new Font("Verdana", Font.PLAIN, 25));
            timer.scheduleAtFixedRate(task, 0, 10);
        } else if (!running) {
            System.exit(0);
        }
    }

    public void makeWalls(int offset) {
        Random rnd = new Random();
        try {
            dirt = ImageIO.read(getClass().getResourceAsStream("/art/dirt block.png"));
            grass = ImageIO.read(getClass().getResourceAsStream("/art/grass block.png"));
            lava = ImageIO.read(getClass().getResourceAsStream("/art/lava block.png"));
            cloud = ImageIO.read(getClass().getResourceAsStream("/art/cloud block.png"));
            undergroundLava = ImageIO.read(getClass().getResourceAsStream("/art/underground lava.png"));
            thumbnail = ImageIO.read(getClass().getResourceAsStream("/art/thumbnail.png"));

        } catch (Exception e) {
            e.printStackTrace();
        }
        if (count == 7) {
            oranges.add(new Orange(offset + 50, 500, s, s, thumbnail));
            for (int i = 1; i <= 14; i++) {
                walls.add(new Wall(offset + i * 50, 600, s, s, grass));
                walls.add(new Wall(offset + i * 50, 650, s, s, dirt));
            }

        }
        else if(count != 10) {
            if (index == 0) {
                for (int i = 1; i <= 14; i++) {
                    walls.add(new Wall(offset + i * 50, 600, s, s, grass));
                    walls.add(new Wall(offset + i * 50, 650, s, s, dirt));
                    if(i == 6) {
                        enemies.add(new Enemy(offset + 350, 550, r, s));
                    }
                }
                set = 0;
            } else if (index == 1) {
                for (int j = 1; j <= 14; j++) {
                    for (int i = 650; i >= 350; i -= 50) {
                        if (i == 350)
                            walls.add(new Wall(offset, i, s, s, grass));
                        else
                            walls.add(new Wall(offset, i, s, s, dirt));
                    }
                    offset += 50;
                    set = 1;
                }
            } else if (index == 2) {
                if(set == 0) {
                    walls.add(new Wall(offset - 50, 500, s, s, cloud));
                    walls.add(new Wall(offset - 200, 400, s, s, cloud));
                    walls.add(new Wall(offset, 300, s, s, cloud));

                    for (int i = 0; i < 9; i++) {
                        walls.add(new Wall(offset + 300 + i * 50, 600, s, s, grass));
                        walls.add(new Wall(offset + 300 + i * 50, 650, s, s, dirt));
                    }
                }else {
                    for (int i = 1; i <= 14; i++) {
                        walls.add(new Wall(offset + i * 50, 600, s, s, grass));
                        walls.add(new Wall(offset + i * 50, 650, s, s, dirt));
                        if(i == 6) {
                            enemies.add(new Enemy(offset + 350, 550, r, s));
                        }
                    }
                }
            } else if (index == 3) {
                for (int i = 1; i <= 2; i++) {
                    walls.add(new Wall(offset + i * 50, 600, s, s, grass));
                    walls.add(new Wall(offset + i * 50, 650, s, s, dirt));
                }
                lavaWalls.add(new LavaWall(offset + 150, 600, s, s, lava));
                lavaWalls.add(new LavaWall(offset + 200, 600, s, s, lava));
                lavaWalls.add(new LavaWall(offset + 250, 600, s, s, lava));
                lavaWalls.add(new LavaWall(offset + 300, 600, s, s, lava));
                lavaWalls.add(new LavaWall(offset + 350, 600, s, s, lava));
                lavaWalls.add(new LavaWall(offset + 150, 650, s, s, undergroundLava));
                lavaWalls.add(new LavaWall(offset + 200, 650, s, s, undergroundLava));
                lavaWalls.add(new LavaWall(offset + 250, 650, s, s, undergroundLava));
                lavaWalls.add(new LavaWall(offset + 300, 650, s, s, undergroundLava));
                lavaWalls.add(new LavaWall(offset + 350, 650, s, s, undergroundLava));
                for (int i = 6; i <= 12; i++) {
                    walls.add(new Wall(offset + 100 + i * 50, 600, s, s, grass));
                    walls.add(new Wall(offset + 100 + i * 50, 650, s, s, dirt));
                }
            }
            for(Wall wall : walls) {
                for(LavaWall lava : lavaWalls) {
                    if(lava.hitBox.getBounds() == wall.hitBox.getBounds()) {
                        lavaWalls.remove(lava);
                    }
                }
            }
            count++;
            index = rnd.nextInt(4);
        }
    }

    public void reset() {
        count = 0;
        index = 0;
        player.x = 200;
        player.y = 150;

        cameraX = 150;
        player.xspeed = 0;
        player.yspeed = 0;
        walls.clear();
        enemies.clear();
        lavaWalls.clear();
        oranges.clear();
        offset = -150;
        makeWalls(offset);
        player.isDed = false;
    }

    public void paint(Graphics g) {

        try {
            super.paint(g);
            Graphics2D gtd = (Graphics2D) g;
            player.draw(gtd);
            for(int i = 0; i < walls.size(); i++) {
                try {
                    walls.get(i).draw(gtd);
                } catch (Throwable e) {

                }
            }for(Enemy enemy : enemies)
                enemy.draw(gtd);
            for(LavaWall lava : lavaWalls)
                lava.draw(gtd);
            g.drawImage(thumbnail, 270, 15, 50, 50, null);
            for(Orange orange : oranges)
                orange.draw(gtd);
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

    @Override
    public void actionPerformed(ActionEvent e) {

    }

    public void keyPressed(KeyEvent e) {

        if (e.getKeyChar() == 'a')
            player.keyLeft = true;
        if (e.getKeyChar() == 'w' || e.getKeyCode() == 32)
            player.keyUp = true;
        if (e.getKeyChar() == 's')
            player.keyDown = true;
        if (e.getKeyChar() == 'd')
            player.keyRight = true;
        if (e.getKeyChar() == 'h')
            Player.keyGlide = true;
        if (e.getKeyChar() == 'j')
            player.keyMega = true;
        if (e.getKeyChar() == 'l')
            player.keyRush = true;

    }

    public void keyReleased(KeyEvent e) {

        if (e.getKeyChar() == 'a')
            player.keyLeft = false;
        if (e.getKeyChar() == 'w' || e.getKeyCode() == 32)
            player.keyUp = false;
        if (e.getKeyChar() == 's')
            player.keyDown = false;
        if (e.getKeyChar() == 'd')
            player.keyRight = false;
        if (e.getKeyCode() == 27)
            System.exit(0);
        if (e.getKeyChar() == 'h')
            Player.keyGlide = false;
        if (e.getKeyChar() == 'j')
            player.keyMega = false;
        if (e.getKeyChar() == 'l')
            player.keyRush = false;
    }
}

and here is the player code
import java.awt.*;
import java.awt.event.*;
import java.io.File;
import java.io.IOException;
import java.util.Iterator;

import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.Clip;
import javax.swing.*;
public class Player {
    GamePanel panel;
    int x;
    int y;
    int width;
    int height;
    double xspeed;
    double yspeed;
    boolean keyLeft;
    boolean keyRight;
    boolean keyUp;
    static boolean keyGlide;
    boolean keyDown;
    boolean keyMega;
    boolean keyRush;
    boolean falling;
    boolean isDeathSound;
    boolean isDed;
    Graphics2D gtd;
    long time = 0;
    Timer timer;
    long start;
    long stop;
    double deaths = 0;
    JLabel death = new JLabel("Deaths: 0");
    Rectangle hitBox;
    long waitTime;
    static Clip clip;
    static boolean onGround;

    Player(){

    }
    Player(int x, int y, GamePanel panel){

        this.panel = panel;
        this.x = x;
        this.y = y;
        width = 50;
        height = 100;
        hitBox = new Rectangle(x, y, width, height);
    }
    public void set() throws Throwable{
        Iterator<Orange> orangeItr = panel.oranges.iterator();
        if(keyLeft && keyRight || !keyLeft && !keyRight) xspeed *= 0.8;
        else if(keyLeft && !keyRight)xspeed--;
        else if(keyRight && !keyLeft)xspeed++;
        if(keyUp && keyGlide) {keyUp = false;}
        if(yspeed > 0)onGround = false;
        if(keyMega && keyGlide) {keyMega = false;}
        if(onGround && keyGlide)keyGlide = false;
        if(xspeed > 0 && xspeed < 0.75)xspeed = 0;
        if(xspeed < 0 && xspeed > -0.75)xspeed = 0;
        if(xspeed > 7 && !keyRush)xspeed = 7;
        if(xspeed < -7 && !keyRush)xspeed = -7;
        if(xspeed > 12 && keyRush)xspeed = 12;
        if(xspeed < -12 && keyRush)xspeed = -12;

        if(yspeed < -12.75) yspeed = -12.75;
        if(keyUp && onGround) {
            jumpSound();
            hitBox.y++;
            onGround = false;
            for(Wall wall : panel.walls) {

                if(wall.hitBox.intersects(hitBox)) {
                    yspeed = -8;

                }
            }hitBox.y--;
        }if(keyMega && onGround) {
            megaSound();
            onGround = false;
            hitBox.y++;

            for(Wall wall : panel.walls) {
                if(wall.hitBox.intersects(hitBox)) yspeed = -12.75;

            }hitBox.y--;

        }
        yspeed += 0.3;
        if(keyGlide && !onGround && yspeed > 1.2) {
            glideSound();
            yspeed -= 0.25;

        }if(keyDown && !onGround) {
            yspeed += 0.4;
        }
        //horizontal collision
        hitBox.x += xspeed;
        for(Wall wall : panel.walls) {
            if(hitBox.intersects(wall.hitBox)) {
                hitBox.x -= xspeed;
                while(!wall.hitBox.intersects(hitBox))hitBox.x += Math.signum(xspeed);
                hitBox.x -= Math.signum(xspeed);
                panel.cameraX += x - hitBox.x;
                xspeed = 0;
                hitBox.x = x;
            }

        }
        for(Orange orange : panel.oranges){
            isDed = false;
            if(hitBox.intersects(orange.hitBox)) {
                isDed = false;
                if(panel.orangeCount != 6) {
                    isDed = false;
                keyGlide = false;
                hitBox.x -= xspeed;
                while(!orange.hitBox.intersects(hitBox))hitBox.x += Math.signum(xspeed);
                hitBox.x -= Math.signum(xspeed);
                panel.cameraX += x - hitBox.x;
                xspeed = 0;
                x = hitBox.x;
                panel.orangeCount++;
                panel.reset();
                panel.orangeLabel.setText(": " +(int)panel.orangeCount + "/7");
                onGround = true;
                isDed = false;
                deathSound();
                }
                else {
                    keyGlide = false;
                    hitBox.x -= xspeed;
                    while(!orange.hitBox.intersects(hitBox))hitBox.x += Math.signum(xspeed);
                    hitBox.x -= Math.signum(xspeed);
                    xspeed = 0;
                    x = hitBox.x;
                    onGround = true;
                    panel.orangeCount++;
                    panel.orangeLabel.setText(": " + (int)panel.orangeCount + "/7");
                    orangeSound();
                    System.out.println("Well done");
                    Thread.sleep(1100);
                    System.out.print("your time is ");
                    System.out.println(panel.timeLabel.getText());
                    System.out.println("you died " + (int)deaths + " time(s)");
                    System.exit(0);
                }
            }
        }

        //vertical collision
        hitBox.y += yspeed;
        for(Wall wall : panel.walls) {
            if(hitBox.intersects(wall.hitBox)) {

                keyGlide = false;
                hitBox.y -= yspeed;
                while(!wall.hitBox.intersects(hitBox))hitBox.y += Math.signum(yspeed);
                hitBox.y -= Math.signum(yspeed);
                yspeed = 0;
                y = hitBox.y;
                onGround = true;
            }
        }
        try {
            for(LavaWall lava : panel.lavaWalls) {
                if(hitBox.intersects(lava.hitBox)) {
                    isDed = true;
                    isDeathSound = true;
                    keyGlide = false;
                    hitBox.y -= yspeed;
                    while(!lava.hitBox.intersects(hitBox))hitBox.y += Math.signum(yspeed);
                    hitBox.y -= Math.signum(yspeed);
                    yspeed = 0;
                    y = hitBox.y;
                    onGround = true;
                    isDed = true;
                    die();

                }
            }
        for(Orange orange : panel.oranges){
                isDed = false;
                if(hitBox.intersects(orange.hitBox)) {
                    isDed = false;
                    if(!(panel.orangeCount == 7)) {

                        keyGlide = false;
                        hitBox.y -= yspeed;
                        while(!orange.hitBox.intersects(hitBox))hitBox.y += Math.signum(yspeed);
                        hitBox.y -= Math.signum(yspeed);
                        yspeed = 0;
                        y = hitBox.y;
                        onGround = true;
                        panel.orangeCount++;
                        panel.reset();
                        panel.orangeLabel.setText(": " +(int)panel.orangeCount + "/7");
                        isDed = false;
                        deathSound();

                    }
                    else if(panel.orangeCount == 7) {
                        keyGlide = false;
                        hitBox.y -= yspeed;
                        while(!orange.hitBox.intersects(hitBox))hitBox.y += Math.signum(yspeed);
                        hitBox.y -= Math.signum(yspeed);
                        yspeed = 0;
                        y = hitBox.y;
                        onGround = true;
                        panel.orangeCount++;
                        panel.orangeLabel.setText(": " + (int)panel.orangeCount + "/7");
                        orangeSound();
                        System.out.println("Well done");
                        Thread.sleep(1100);
                        System.out.print("your time is ");
                        System.out.println(panel.timeLabel.getText());
                        System.out.println("you died " + (int)deaths + " time(s)");
                        Thread.sleep(4500);
                        System.out.println("press v to exit");
                    }
                }
            }
        }catch(Throwable e) {

        }

        panel.cameraX -= xspeed;
        y += yspeed;
        hitBox.x = x;
        hitBox.y = y;
        //death code
        if(y > 800) {
            isDed = true;
            onGround = true;
            die();
        }enemyScript();
    }void enemyScript() {
        for(Enemy enemy : panel.enemies) {
            if(hitBox.intersects(enemy.hitBox)) {
                isDed = true;
                isDeathSound = true;
                keyGlide = false;
                hitBox.y -= yspeed;
                while(!enemy.hitBox.intersects(hitBox))hitBox.y += Math.signum(yspeed);
                hitBox.y -= Math.signum(yspeed);
                yspeed = 0;
                y = hitBox.y;
                onGround = true;
                isDed = true;
                die();
            }
        }
    }
    void jumpSound(){
        soundEffect("res\\musicAndSoundEffects\\jump.wav");
    }void megaSound(){
        soundEffect("res\\musicAndSoundEffects\\megajump.wav");
    }void glideSound(){
        soundEffect("res\\musicAndSoundEffects\\shorterGlide.wav");
    }void deathSound(){
        if(isDed) {
        soundEffect("res\\musicAndSoundEffects\\death.wav");
        }else if(!isDed) {
            soundEffect("res\\musicAndSoundEffects\\orangeSound.wav");
        }
    }void notDie(){
        stopSound();
    }
    void orangeSound(){
        soundEffect("res\\musicAndSoundEffects\\orangeSound.wav");
    }void stopSound(){
        clip.stop();
    }
    void die() {
        deathSound();
        panel.reset();
        deaths++;
        death.setText("Deaths: " + (int)deaths);
        onGround = true;
    }
    static void soundEffect(String filePath) {
        try {
            File file = new File(filePath);
            AudioInputStream sound = AudioSystem.getAudioInputStream(file);
            clip = AudioSystem.getClip();
            if(onGround && !filePath.equals("res\\musicAndSoundEffects\\shorterGlide.wav")) {
                clip.open(sound);
                clip.setFramePosition(0);
                clip.start();
            }else if(filePath.equals("res\\musicAndSoundEffects\\shorterGlide.wav")) {
                clip.open(sound);
                clip.setFramePosition(0);
                clip.start();
            }

        }catch(Exception e) {
            JOptionPane.showMessageDialog(null, e.getStackTrace());
        }
    }
    public void draw(Graphics2D g) {
        g.setColor(Color.BLACK);
        g.fillRect(x, y, width, height);
    }
}

for anyone who knows the answer to this. Please comment the problem and not the answer so that I can learn. thanks
Answer: In broad strokes:
Because you're using a java.util.Timer and it is just the wrong tool for the job.
Let's say you want a game to run smoothly, but not slam the CPU of even the most powerful machine imaginable, and also to run as well as it can if the system it runs on is insufficient (that last one is important; even if a system is plenty powerful to run the game, sometimes you get a 'blip' of underperformance because of other stuff the computer was doing, such as when you plug in a USB stick and all sorts of driver shenanigans occurs in the background).
To accomplish these things, you need two crucial properties:

Timing related to the gameplay (such as, if one of the players is moving northbound, the amount of movement that occurs) should be based entirely on 'time passed', and be unaffected by the amount of time required to actually redraw parts of the screen, nor should it be affected by how many 'loops' of screen refresh you manage to squeeze into a single second.
Timing related to rendering needs to go fast as possible, but needs a breaking system: BEFORE you begin the task of 'update game state then render the state to the screen', you should calculate the timing at which that should all be done. Then, once you are done with one loop of the game (adjust game state and rendering all that), you check whether you 'beat the clock'. If you did, THEN wait. In other words, instead of asking a timer to wait a fixed amount of time before doing the next game loop, the amount of time to wait needs to take into account you took some time to adjust the game state and render things.

Here's some example code of a basic game loop. Just use it for inspiration, it doesn't fit verbatim into the code you already have.
class Example {
  private static final long NANOS_PER_MILLI =
    TimeUnit.MILLISECONDS.toNanos(1);
  private static final long NANOS_PER_FRAME =
    TimeUnit.SECONDS.toNanos(1) / 60;
  private static final long ADJUST_EXPECTATIONS_TRESHOLD =
    NANOS_PER_FRAME * 5;

  private long start = System.nanoTime();
  private long frameCounter = 0;

  public void run() {
    while (running) {
      gameLoop();
    }
  }

  void gameLoop() {
    frameCounter++;
    long target = startTime + NANOS_PER_FRAME * frameCounter;

    // target now holds the exact time at which point we should
    // be done with this game loop.

    recalcGameState();
    redrawStuff();

    long end = System.nanoTime();
    long delta = target - System.nanoTime();

    if (delta > 0) {
      // We had time to spare!
      // Let's sleep most of that away, giving us
      // a little bit of leeway (5msec worth) in case the system is
      // going to blip on us.

      // Also, CPU waiting is generally granularized at millis,
      // so let's convert to millis first, and be 5 millis 'early'

      long millisToWait = (delta / NANOS_PER_MILLI) - 5;
      if (millisToWait > 0) sleep(millisToWait);
    else if (-delta > READJUST_EXPECTATIONS_TRESHOLD) {
      // Rats - we missed the boat, and by a lot!
      // we are over 5 frames behind, let's readjust
      // expectations and slow down the game logic, too:

        start = end;
        frameCounter = 0;
      }
    }
  }
}

The above code will reliably render at 60 gameloops every second, (that 5msec leeway will adjust itself on the next loop, as that would just mean you get there even faster and thus wait longer) - and will 'catch up' on a blip for at most 5 frames (catching up a ton of frames after a long blip is usually not a good idea, so we limit the catchup mechanism).
Usually when you're forced into readjustment you may want to adjust the game logic's clocks as well (it is a bit unfair for a player to slam into a wall that was way ahead of them because their CPU just decided to doze off for a second, unlikely as that is, whilst they couldn't steer or even see anything!) - so, have a ball with gameStart.
NB: nanoTime is used and not System.currentTimeMillis(), because System.CTM represents the system clock, which means it will drastically change when the user resets their time - modern computers update their clocks by way of a network time server, so this happens all the time (heh), and means that looking at it for frame timing is a bad idea. From time to time the game will do mad glitching when the NTP daemon adjusted the clocks. nanoTime is more or less reflecting 'CPU uptime' and does not get modified when the system clock is adjusted.
NB2: If you need 'frameCounter' to represent game turns, you can't just reset it like this; instead, figure out how many frames you want to adjust by, and then add that many NANOS_PER_FRAME to 'start' instead.


Are repeats, segnos, and coda markings "bad style" in marching band music?

Question: Nearly all marching band music (not classic marches, mind) that I've seen in my five years of performing experience has been written without any kind of repeat signs. However, I've recently seen some marching arrangements of jazz standards that did in fact use repeat bars.
Printing in a "through-composed" manner seems to make more sense to me, as it would be easier for performers to annotate the sheet music as needed, and easier to coordinate drill charts with measure numbers.
Are these markings actually more common than I think?
Comment: What you haven't memorized your part? (seriously- all top marching bands do)
Answer: I'm glad you excepted classic marches, which invariably have at least one BIG repeat, a DC after the trio section.
Yes, I can see the logic in 'writing it out' when complex drill moves are involved.  I can't speak for the House Style of any particular publisher, but if you're asking for advice how YOU should score music I'd borrow Einstein's advice to make notation 'as simple as possible, but no simpler'.   Use the minimum amount of ink that conveys all necessary information.  This will depend on the music content and on the circumstances where it's to be performed.  In movie soundtrack recording sessions we write everything out. Where music won't have to be sight-read we can trade a convoluted repeat structure for economy of paper.
Comment: OK.  But I guess marching bands get a fair amount of rehearsal?   Instant sight-reading may not be THAT much of a priority!
Comment: I didn't think about a written-out section being easier to understand quickly. For the arrangement I'm working on right now, I think you've just sold me on writing it that way, instead of the (perhaps unjustifiably short) D.S. al coda.
Comment: They do get a good amount of rehearsals, and private practice is also encouraged to memorize the music. However, I'm thinking that if you have to spend less cognitive energy following repeat marks around, the work of memorization might be easier (and my arrangement could be more marketable).
Comment: University bands (and some high school bands too) will usually play a different show every week, so there might be a little more of a parallel to session musicians than to, say, an orchestra or wind band.
Answer: Another reason for durchgedruckt music is that often, the repeats have slightly different orchestration or ornamentation or the like. Sousa's repeat of the trio in Semper Fidelis is an example; new instruments are added at each section.
DCs save paper but often make reading complicated. A few repeats with jumps and different instructions each time through (1st alletro, 2nd tempo di PBS documentary) become difficult.
Comment: "Durchgedruckt" - I *KNEW* there was a real word for it!


Variance and bias

Question: Consider the mean squared error $$\text{MSE}(x_0) = E_{T}(f(x_0)-\hat{y}_0)^{2}$$ $$ =  E_{T}(\hat{y}_0-E_{T}(\hat{y}_0))^{2}+(f(x_0)-E_{T}(\hat{y}_0)^{2})$$
Is the first term after the equal sign in the second line basically saying the following: Get an estimate and find its deviation from the average estimates over all training samples. The estimate $\hat{y}_0$ will depend on the training set which is why we take the expected value.
Also is $E_{T}(\hat{y}_0)$ constant?
Comment: You probably mean $MSE(y_0)$, not of $x_0$. Also, not sure what the expectation $E_T$ stands for -- what is the probability space over which you are integrating? Is your $x$ random or fixed?
Answer: Your intuition seems right, but for clarity, I would rewrite your equations as follows,
\begin{align}
\text{MSE}(\hat{y})
 &= E\left[(y-\hat{y})^2\right] \\
 &= \underbrace{E\left[(\hat{y}-E[\hat{y}])^2\right]}_{\text{Var}[\hat{y}]} + \underbrace{(y-E[\hat{y}])^2}_{\text{Bias}^2}
\end{align}
Here $y$ is the actual value of whatever we are estimating, and $\hat{y}$ is its estimator (the value of $\hat{y}$ will depend on the data).
The first term is the variance of the estimator (which is more or less what you said). Also, $E[\hat{y}]$ is non-random (this may be what you meant by constant).

Variance and bias


Change Underscore to Dash in Apache Header

Question: I have an application that runs behind Apache 2.4.27 that authenticates users with mod_auth_cas. This module stores attributes about the authenticated user in headers. I'm having issues with one header that's being stripped because it has an underscore in it (X-CAS-email_primary). After doing some searching, I added the code snippet below to the VirtualHost configuration.
SetEnvIfNoCase X-CAS-email_primary ^(.*)$ fix_header=$1
RequestHeader set X-CAS-email-primary %{fix_header}e env=fix_header

In case it wasn't obvious, I'm trying to change X-CAS-email_primary to X-CAS-email-primary. This code is partially working. The new header does appear but it has no value.
["HTTP_X_CAS_EMAIL_PRIMARY"]=> string(0) ""

I've confirmed that X-CAS-email_primary is set, so I'm not quite sure what's going on. Any suggestions?
Thanks!
Answer: The value of a RequestHeader set supports expressions, and expressions include the req (or http) function, which gives you the value of request headers. So this one directive should do what you want:
RequestHeader set X-CAS-email-primary "expr=%{req:X-CAS-email_primary}"

You have to dig deep into the documentation to find this kind of thing, but it's there.
Not sure why your configuration didn't work, but I guess the SetEnvIfNoCase is evaluated after the RequestHeader. The docs don't make it easy to figure that out.


Find the Fourier series of a function , determining whether it converges pointwise/uniformly

Question: I've been trying to solve an exercise from a test. I could use your help :)

Question
Consider the function:
$g(x) =  \frac{\pi}{4} $ for $x\in [0,\pi ]$ and $ -\frac{\pi}{4}$ for $x\in (-\pi,0 )  $. Let $f(n)$ be $g(x)$'s  periodic continuation.
$S_N f(x) = A_0 +\sum_{n=1}^{N}A_n \cos{nx}+i\sum_{n=1}^{N}B_n \sin{nx}$ where $A_0 = \hat{f}(0), A_n = \hat{f}(n)+\hat{f}(-n), B_n = \hat{f}(n)-\hat{f}(-n)$.
Note: The Fourier series of $f$ is $\lim_{N \to \infty } S_Nf$

Find $S_Nf$.
Determine if $S_Nf(x)$ pointwise converges for $x\in \left[-\pi,\pi \right]$.
Determine if $S_Nf(x)$ uniformly converges in $ \left[-\pi,\pi \right]$.
Prove $\lim_{N \to \infty} S_Nf \left( \frac{1}{\pi N}\right) = \int_{0}^{1} \frac{\sin{\left( \frac{t}{\pi}\right)}}{2t} \ dt $

My attempt
Sol for 1:
$\hat{f}(n)= \frac{1}{2in}$ for $n\in \mathbb{Z}_{odd} $ , else it's $0$.
Hence, $S_N f (x) = \sum_{k=0}^{K}\frac{1}{(2k+1)} \sin{((2k+1)x)}$ where $K=\left\lfloor \frac{N}{2}\right\rfloor$

Sol for 3:
$S_N f (x)$ is a continuous function, therefore if $S_Nf$ uniformly converges we know it has to be to $f$. But $f$ isn't continuous. Therefore we will conclude
$S_Nf$ does not uniformly converges.

I would appreciate your help in 2 and 4 (if you do please be formal so I can understand better). Thank you!
Answer: Solution to (2)
There is a Theorem by Dirichlet that states:

"Given a $2\pi$-periodic function $f$ which is generally continuous and whose derivative is generally continuous in $[-\pi,+\pi]$ (by a generally continuous function I mean that it has finitely many dicontinuities in its domain and that each on of them is a jump dicontinuity), its Fourier Series converges uniformly to $f$ in every sub-interval in which $f$ is continuos and converges to the mean value of the left and right limits of $f$ at each point of discontinuity of $f$"

From this theorem it follows immediately that $S_N(f)$ converges pointwise to some function in $[-\pi,+\pi]$ but that function is not exactly $f$ (e.g, $\lim_{N \to +\infty} S_N(f)(0)=0$). You could also use this theorem to answer point (3).
Solution to (4)
First of all notice that we can compute $\lim_{N\to+\infty}S_N(f)(\frac{1}{\pi N})$ by computing  $\lim_{N\to+\infty}S_{2N+1}(f)(\frac{1}{\pi (2N+1)})$, since it is an injective change of variable. Then, having written the expression for this limit explicitly:
$$ \lim_{N\to+\infty} \sum_{n=0}^{N} \frac{1}{2n+1} \sin(\frac{2n+1}{\pi (2N+1)}) $$
we let $x_{n}=\frac{2n+1}{\pi (2N+1)}, \hspace{3mm} n=0,...,N$ and $\Delta x = \frac{2}{\pi (2N+1)}$ in order to obtain:
$$ \lim_{N\to+\infty} \sum_{n=0}^{N} \frac{\sin(x_n)}{2x_n} \Delta x $$
but this Riemann Sum converges exactly to the desired integral.
Comment: So, $S_Nf(x)$ coverges to $f$ for $x\in (0,\pi) \cup (-\pi,0)$ and for $x=0,-\pi ,\pi$  $S_Nf(x)$ coverges to $0$?
Comment: I think you meant "First of all notice that we can compute $\lim_{N\to+\infty}S_N(f)({1\over \pi N}) $" instead of $\lim_{N\to+\infty}S_N(f)(x)$
Comment: Do you compute $\lim_{N\to+\infty}S_{2N+1}(f)(\frac{1}{\pi (2N+1)})$ instead of $\lim_{N\to+\infty}S_N(f)({1\over \pi N}) $ because it's more convenient, right?
Comment: @Lior As for your first question: yes. As for your second question: yeah thanks for pointing it out I will change it immediately . As for your third question: yes, because it is easier to reconstruct the Riemann sum.
Comment: It's ok, no big deal :) The idea is what matters. It's nice that you noticed that it's a Riemann sum.
Comment: Sorry to bother you, but why is there a $2$ in $\lim_{N\to+\infty} \sum_{n=0}^{N} \frac{\sin(x_n)}{2x_n} \Delta x$. I think it's  without the $2$ , I mean $\lim_{N\to+\infty}S_{2N+1}(f)(\frac{1}{\pi (2N+1)})=\lim_{N\to+\infty} \sum_{n=0}^{N} \frac{\sin(x_n)}{x_n} \Delta x$
Comment: @Lior Actually it is correct, the problem is that I missed a 2 in the definition of $\Delta x$ (see answer) otherwise you don't have $x_{n+1}-x_n=\Delta x$. Again, my bad.


Change of variable in Bertsekas nonlinear programming chapter 1.3

Question: I'm trying to understand how a very simple change of variables works for the following gradient descent equation.
\begin{align}
x^{k+1} = x^k - \alpha^kD^k \nabla f(x^k)
\end{align}
This is supposedly viewed as a scaled version of steepest descent and they prove it by setting:
\begin{align}
S = (D^k)^{1/2}
\end{align}
They then use this to consider the transformation of variables
\begin{align}
x = Sy
\end{align}
The minimization problem gets rewritten as
\begin{align}
&\text{minimize } h(y) \equiv f(Sy) \\
&\text{subject to } y \in \mathbb{R}^n
\end{align}
And finally they say they can write this as follows:
\begin{align}
y^{k+1} = y^k - \alpha^k \nabla h(y^k)
\end{align}
Where does $D^k$ go? My work so far is simply plugging in $Sy$ for $x$ and then multiplying by the inverse as follows which gets me the following:
\begin{align}
Sy^{k+1} &= Sy^k - \alpha^k D^k \nabla f(Sy^k) \\
S^{-1}(Sy^{k+1}) &= S^{-1}(Sy^k - \alpha^k D^k \nabla h(y^k)) \\
y^{k+1} &= y^k - \alpha^k S^{-1}D^k \nabla h(y^k) \\
y^{k+1} &= y^k - \alpha^k D^{-1/2}D^k \nabla h(y^k) \\
\end{align}
Where do I find my other $D^{-1/2}$ to make $D^k$ go to 1?
Edit: Here is the solution for anyone who needs it:
We need the $D = S^2$ and $\nabla h(y^k) = S\nabla f(Sy^k)$
\begin{align}
Sy^{k+1} &= Sy^k - \alpha^k S^2 \nabla f(Sy^k) \\
Sy^{k+1} &= Sy^k - \alpha^k S \nabla h(y^k) \\
S^{-1}(Sy^{k+1}) &= S^{-1}(Sy^k - \alpha^k S \nabla h(y^k)) \\
y^{k+1} &= y^k - \alpha^k \nabla h(y^k) \\
\end{align}
Answer: You need to write the gradient of $h(y) = f(Sy)$ wrt $y$:
$\nabla_y h(y) = S^T \nabla_x f(Sy) = S \nabla_x f(Sy)$ ($S$ seems to be symmetric?)
Could be where you had a missing $S$.
Comment: Right I see now. $S^2 = D$ and since $\nabla h(y) = S'\nabla f(Sy)$ as you said in your initial comment. That clears everything up thanks for the help!
Comment: That makes sense, but wouldn't that give my last step to be $y^{k+1} = y^k - \alpha^k D^{-1/2}D^k D^{1/2} \nabla h (y^k)$. But yes S is symmetric and $D = D^{1/2}D^{1/2}$
Comment: From $x^{k+1} = x^k - \alpha^kD^k \nabla f(x^k)$, we have $Sy^{k+1} = Sy^k - \alpha^k S^2 \nabla f(Sy^k)$, then $Sy^{k+1} = Sy^k - \alpha^k S \nabla_y h(y^k)$, then compose by $S^{-1}$.


Logistic regression: age covariate within age group strata

Question: I am using logistic regression to calculate the OR of falling using certain predictors (e.g. strength), age is also a confounder. The effect of age appears linear - risk of falling incrementally increases with age. Age also has a weak/negative correlation with strength.
When age is added as a covariate to the full group (n=2000), strength is significant. However, I have been advised to consider splitting my group into different age groups i.e. 2-3 age group strata. If I do this, firstly groups are uneven (e.g. old n=1500, very old n=500) and secondly strength loses significance in the old group; age also loses significance in both groups.
I struggle to understand the difference between age as a covariate and age group strata. I have seen the following post - Cox regression: age covariate within age group strata - but I am struggling to apply this to my analysis as it is a retrospective observational cohort study and the effect of age is linear.
I would be really grateful if anyone could explain (simply) how the analysis of age as a covariate and age group strata differs? And also why I might be losing significance for strength when using age strata?
Answer: The idea of losing information about continuous age is in general absurd and will result in meaningless analyses with unaccounted-for outcome heterogeneity.  The only reason to ever stratify on age in a Cox model setting is if you have serious non-proportional hazards for age that you want to partially correct for.  On top of stratification you would include smooth nonlinear flexible effects for age (e.g., using a regression spline).  But there are perhaps better solution.  You might for example find that an accelerated failure time model fits better in some situations, and you might include an age by time interaction as a special time-dependent covariate instead of stratifying.
Comment: Frank, thank you so much for taking the time to respond to my question. Apologies for being unclear in my post, I have now edited it, but my study is a large, observational cross-sectional cohort study and I am using logistic rather than Cox regression. The suggestion to stratify is based on theoretical/biological reasons i.e. a 60y old and an 85y old have very different body composition characteristics etc. However, the risk increase for every extra 5y of age seems roughly proportionate (30-40% increase with every 5 years).
Comment: I would be so grateful if you could explain how adding age as a covariate and analysing the full group (n=2000)  differs from analyses stratified by age e.g. old (n=1500) and very old (n=500). Thank you again
Comment: Splitting by age pretends that age is a binary variable.  That is not the case.  Such splitting is invalid as it assumes a piecewise flat relationship with outcome with a sharp discontinuity at the cut point.  Adjust for age flexibly and continuously (using e.g. a regression spline) and forget about stratification.  Only stratifying by age will result in residual outcome heterogeneity that is left unexplained.