TeamCity upgrade fails with "Can't take exclusive lock when db lock is not held"

Question: When upgrading to TeamCity 2023.05 (using the Docker image), I received the error "Can't take exclusive lock when db lock is not held" and the detailed instruction:

java.lang.IllegalStateException: Can't take an exclusive lock when db lock is not held
    at jetbrains.buildServer.serverSide.db.Heartbeat.tryGetExclusiveLock(Heartbeat.java:726)
    at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.tryGetExclusiveLock(TeamCityDatabaseManager.java:857)
    at jetbrains.buildServer.maintenance.StartupProcessor.lockDatabaseExclusively(StartupProcessor.java:860)
    at jetbrains.buildServer.maintenance.StartupProcessor.doUpgradeRequired(StartupProcessor.java:773)
    at jetbrains.buildServer.maintenance.StartupProcessor.checkUpgradeRequired(StartupProcessor.java:826)
    at jetbrains.buildServer.maintenance.StartupProcessor.doDatabaseChecking(StartupProcessor.java:1377)
    at jetbrains.buildServer.maintenance.StartupProcessor.access$1000(StartupProcessor.java:1410)
    at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:12)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
    at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:779)
    at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:1165)
    at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1100)
    at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1120)
    at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
    at java.base/java.lang.Thread.run(Thread.java:833)

I spent a lot of time trying to get to the bottom of this, as I initially thought it was something wrong with the actual database, but could not find any locks.
Answer: The solution was to drop the records in the db_heartbeat table in the teamcity database.
Check if there are any records:
select * from db_heartbeat;
[...]

Remove all records:
TRUNCATE TABLE db_heartbeat;

Verify that they were dropped:
select * from db_heartbeat;
[...]

Once done, you should be able to restart TeamCity and redo the migration.


Barcode is not getting scanned through my scanner machine in mvc.net application

Question: I have generated my barcode image through code 39 when i going to scan image on my desktop then its not getting scanned.And Also not working when i print that Barcode image and scan after.
public Byte[] getBarcodeImage(string barcode, string file)
{
    try
    {
        BarCode39 _barcode = new BarCode39(); int barSize =25;
        string fontFile = HttpContext.Current.Server.MapPath("~/fonts/FREE3OF9.TTF");
        return (_barcode.Code39(barcode, barSize, true, file, fontFile));
    }
    catch (Exception ex) { }
    return null;
}

Comment: Where is the code?
Comment: I have a barcodecs as follows    public Byte[] getBarcodeImage(string barcode, string file)
        {
            try
            {
                BarCode39 _barcode = new BarCode39();
                int barSize =25;
                string fontFile = HttpContext.Current.Server.MapPath("~/fonts/FREE3OF9.TTF");
                return (_barcode.Code39(barcode, barSize, true, file, fontFile));
            }
            catch (Exception ex)
            {

            }
            return null;
        }
Comment: It s generate image but generated image is not getting scanned.Please help me to solve this problem.Thank you in advance
Comment: What is the error message you are getting?
Comment: I am not getting any error BarcodeImage is getting successfully,But not getting scanned through Scanner.
Comment: when i printed out this all images and trying to scan those images but i fail.
Answer: According to  this article and the comments within you first need to increase the bar size. I set mine at 64. Then you need to add * character to beginning and end of barcode image like this: return (_barcode.Code39($"*{barcode}*", barSize, true, file, fontFile)); This should work and when you scan you get your barcode without the * characters.


Interpreting random effects estimate in logistic regression with binary predictor

Question: I've run a model in R as follows:
model1 <- glmer(outcome ~ predictor + (1 | groupID), data, family = binomial())

Like outcome, predictor is a binary variable. My understanding is that this model would only give me the random effects estimate of groups where predictor = 0 (because 0 would be the intercept), so I also ran the following to include a slope term and hopefully estimate the variance of groups where predictor = 1:
model2 <- glmer(outcome ~ predictor + (predictor | groupID), data,
                family = binomial())

I ran a dotplot using the 'lattice' package and was confused about seeing the same groups in both intercept and slope plots. I don't understand what the slope estimate plot is telling me.
How do I get a random effects estimate for all groups or those for which predictor == 1?
I think a problem might be that all individuals within each group have the same value of predictor.
Answer: Quite a few things going on at once here: your first model estimates random effects for all groupID. What it prints in the summary is the estimated standard deviation across blocks, which is of interest in the marginal model.
If you want to get individual random effect estimates, you can use lme4::ranef(model1) to extract them. This will return the random intercepts, one per groupID in your input.
Your second model indeed adds a random "slope" on predictor (assuming it's coded 0/1), but the marginal estimate will still be the same for all observations. If you want to plot the individual random effects you'll have to extract  them as above.
Comment: Thank you for answering my question. I have another question related to this, but I thought I should probably not ask it here. In case you are up for answering another, it's here: https://stats.stackexchange.com/questions/629081/why-do-my-random-effects-estimates-not-match-the-data


Compute the terms $c_2,c_3,c_4$ of a recursively defined relation $c_k=c_{k-1}+kc_{k-2}+1$ ...

Question: Could anyone please tell me if my steps and solution is correct or should I change something?
Compute the terms $c_2,c_3,c_4$ of a recursively defined relation
\begin{align}
c_k &=c_{k-1}+kc_{k-2}+1 & \text{ for all integers  $k \ge 2$}
\end{align}
where initial conditions are $c_0=1$ and $c_1=2$.

Working:
\begin{align}
k = 2,\\
c_2 &= c_1 + 2c_0 + 1\\
&= 2 + 2(1) + 1\\
&= 5\\
\\
k = 3,\\
c_3 &= c_2 + 3c_1 +1\\
&= 5 + 3(2) +1\\
&= 12\\
\\
k = 4,\\
c_4 &= c_3 + 4c_2+1\\
&= 12 + 4(5) + 1\\
&= 33
\end{align}
Comment: Absolutely correct answer.
Comment: Right -- the problem is that the "for all integers $k \ge 2$ is in the wrong place -- it should modify the recurrence. That mis-led me. I've slightly re-formatted the question to make this a little clearer (to me, anyhow).
Comment: @jeana You can also use Wolfram Alpha to check your answer. Replacing $c_k$ with $g(n)$, your answer is [correct](https://www.wolframalpha.com/input/?i=g(n)%3Dg(n-1)%2Bn*g(n-2)%2B1,+g(0)%3D1,+g(1)%3D2). In addition, searching the first few terms on [OEIS](https://oeis.org/A186739) gives some more terms.
Answer: Perfect.
You did exactly what you were asked to do with sufficient details.
If you want to spend more time on the problem, you may try to find a formula for $C_n$ in terms of $n$


Questions about fps for airsoft

Question: I'm considering getting the matrix sr25 full metal that shoots 400-450 fps, what is a good minimum engagement distance for this? Me and my friend are playing together so we don't have a set of rules to go by.
Thanks!
Answer: It depends on who you ask, but you're looking at a consensus minimum engagement distance (MED) of around 15 metres and recommended engagement distance (RED) of around 20 metres, and in semi-auto mode only.
A variety of posts in this forum's thread, using 0.20g BB ammunition at 450fps, mention variously

50 feet (15 metres) (Utah)
20 metres (Brazil)
10 metres (Indonesia)
40 feet (12 metres) (Texas)
20 metres (Norway)

Websites for airsoft associations in other countries describe similar limits, MED 15-20 metres and RED 20-25 metres. You may be best advised to start at the high end of the range and work down as you develop comfort with being hit and judgement of appropriately safe distances.

Questions about fps for airsoft


Commutation of differentiation with any linear map

Question: Consider $A$ and $B$ are two linear operators in finite-dimensional vector space. So to commute, they should share something special. For matrix $A$ to commute with all other is quite restrictive: it would only true for multiply of identity.
However differential is also linear operator and it commutes with any matrix $A$:
$$
f(x) = A\phi(x), [D_{(x_0)}A\circ\phi(x)](h) = A[D_{(x_0)}\phi(x)](h).
$$
This is quite a puzzle for me from this perspective.
Answer: The differentiation map doesn't commute with every other linear map. Let $V$ denote the vector space of differentiable functions from $R \to R$. Define $D : V \to V$ by $$Df = f'$$ and define $ T : V \to V$ by $$Tf = xf(x)$$
So, for example, $T(x^2) = xx^2 =x^3$. It's a bit of an abuse of notation.
As you can see that $$(DT)(x^2)= 3x^2$$ Whereas $$(TD)(x^2) = 2x^2$$
Comment: Thank you! Nice example
Answer: First of all, the space of differential forms $\Omega^*(M)$ on a manifold $M$ is not a finite dimensional vector space: it is a finite rank $\mathcal{C}^{\infty}(M)$-module, and an infinite dimensional vector space over $\Bbb R$.
Moreover, in this specific context, the exterior differential $d\colon \Omega^*(M)\to \Omega^*(M)$ does not commute with all linear operators.
For instance, if $X$ is a vector field, the interior multiplication $\iota_X$, given by
$$
\iota_X\omega = \omega(X,\cdot,\ldots,\cdot)
$$
does not commute with $d$ for a generic $X$.
In your example, the matrix you consider has constant coefficient: it is not surprising that $d$ commutes with it since $d$ is a differential operator.
We could expect a sort of Leibniz rule, like
$$
d(A\phi) = dA . \phi + A d\phi,
$$
and $A$ being with constant coefficients, we would have $dA=0$
(caution, what I said here is not totally rigorous).
Comment: Thank you, Leibnitz rule intuition clarify things for me.
Comment: @EvgeniiEgorov You're welcome. I just thought of a better example: taking a function $f\in \mathcal{C}^{\infty}(M)$, the operator $L$ given by the pointwise multiplication by $f(x)$ is linear, but clearly, $d(L(\omega))= df\wedge \omega + fd\omega$, so that $d\circ L - L \circ d = df\wedge \cdot$.
Comment: Sorry, could you please a bit elaborate on notation? I miss how $L(\omega)$ and $f$ related. Thank you.
Comment: @EvgeniiEgorov $L(\omega) = f\omega$, the pointwise multiplication, given by $(f\omega)_x = f(x)\omega_x$.
Answer: It's because the derivative of a linear map $T$ is again just the linear map $T$. This is because the derivative is the best linear approximation, but if you map is already linear, then there is no need to approximate it. For example, in one dimension, this is why $T:\mathbb{R}\to \mathbb{R}$ by $T(x)=\lambda x$ has derivative $\lambda$.
Comment: Nice perspective thank you!


Corollary of Vitali's theorem from measure theory

Question: Definition: Suppose $E\subset \mathbb{R}$ and and let $K=\{I_{\alpha}\}_{\alpha\in \Omega}$ be a family of closed nonempty intervals in $\mathbb{R}$. We say that $K$ is Vitali covering of $E$ if for any $x\in E$ and for any $\epsilon>0$ there is $I=I_{x,\epsilon}$ such that $x\in I$ and $\mu(I)<\epsilon$, where by $\mu$ I mean Lebesgue measure on the real line.

Vitali's theorem: Let $E$ be a bounded subset of $\mathbb{R}$ and let $K$ is a Vitali covering of $E$. Then there is countable family of
  pairwise disjoint $\{I_n\}_{n=1}^{\infty}\subset K$ such that
  $$\mu^*(E- \sqcup _{n=1}^{\infty}I_n)=0$$

Corollary: Under the assumptions of the above theorem one can find finitely many pairwise disjoint intervals $\{I_k\}_{k=1}^{n}\subset K$ such that $$\sum \limits_{k=1}^n\mu(I_k)>\frac{1}{2}\mu^*(E).$$
Can anyone explain to me how the corollary follows the theorem, please? It is not so clear to me .
EDIT: I was able to derive the following inequality: $$\mu^*(E)\leq \sum \limits_{k=1}^{\infty}\mu(I_k)$$
Answer: Since $I=\cup_{n=1}^\infty I_n$ is measurable
$\mu^* E = \mu^*(E \setminus I) + \mu^*(E \cap I) \le \mu^* I = \mu I$.
Since $\lim_{N \to \infty} \cup_{n=1}^N \mu I_n = \mu I$ we can find some $N$ such that
$\cup_{n=1}^N I_n  > { 1\over 2} \mu I \ge {1 \over 2} \mu^* E$.
Comment: Thanks for catching that, and yes regarding continuity of measure.
Comment: Thanks a lot for your help! I really appreciate it!
Comment: Glad to be able to help :-).
Comment: Well, if $\mu I $ is infinite, then $\sum_{n=1}^N \mu I_n \to \infty$ so you can always find some $N$ so that $\sum_{n=1}^N \mu I_n$ is greater than any number you want. So yes, there is a bug in the proof above, but if $\mu I = \infty$ then the corollary is immediate.
Comment: Hmm. Really nice! Probably you meant $\sum_{n=1}^N\mu(I_n)\to \mu(I)$ because Lebesgue measure is continuous, right?
Comment: Sorry for the belated remark but the conclusion which you wrote does not makes sense if $\mu(I)=\infty$. So i guess firstly we need to show that $\mu(I)<\infty$, right?


versioncontrol.GetAllTeamProjects() returning 0

Question: I'm working on a project where I'll have a C# application search all TFS project's .cs files for a string or string patterns. I found some code that looks similar to what I was looking to do here.
namespace TFSSearch
{
class Program
{
    static string[] textPatterns = new[] { "void main(", "exception", "RegisterScript" };  //Text to search
    static string[] filePatterns = new[] { "*.cs", "*.xml", "*.config", "*.asp", "*.aspx", "*.js", "*.htm", "*.html",
                                           "*.vb", "*.asax", "*.ashx", "*.asmx", "*.ascx", "*.master", "*.svc"}; //file extensions

    static void Main(string[] args)
    {
        try
        {
            var tfs = TfsTeamProjectCollectionFactory
             .GetTeamProjectCollection(new Uri("https://{tfsserver}/tfs"));

            tfs.EnsureAuthenticated();

            var versionControl = tfs.GetService<VersionControlServer>();

            StreamWriter outputFile = new StreamWriter(@"C:\Find.txt");
            var allProjs = versionControl.GetAllTeamProjects(true);
            foreach (var teamProj in allProjs)
            {
                foreach (var filePattern in filePatterns)
                {
                    var items = versionControl.GetItems(teamProj.ServerItem + "/" + filePattern, RecursionType.Full).Items
                                .Where(i => !i.ServerItem.Contains("_ReSharper"));  //skipping resharper stuff
                    foreach (var item in items)
                    {
                        List<string> lines = SearchInFile(item);
                        if (lines.Count > 0)
                        {
                            outputFile.WriteLine("FILE:" + item.ServerItem);
                            outputFile.WriteLine(lines.Count.ToString() + " occurence(s) found.");
                            outputFile.WriteLine();
                        }
                        foreach (string line in lines)
                        {
                            outputFile.WriteLine(line);
                        }
                        if (lines.Count > 0)
                        {
                            outputFile.WriteLine();
                        }
                    }
                }
                outputFile.Flush();
            }
        }
        catch (Exception e)
        {
            string ex = e.Message;
            Console.WriteLine("!!EXCEPTION: " + e.Message);
            Console.WriteLine("Continuing... ");
        }
        Console.WriteLine("========");
        Console.Read();
    }

    // Define other methods and classes here
    private static List<string> SearchInFile(Item file)
    {
        var result = new List<string>();

        try
        {
            var stream = new StreamReader(file.DownloadFile(), Encoding.Default);

            var line = stream.ReadLine();
            var lineIndex = 0;

            while (!stream.EndOfStream)
            {
                if (textPatterns.Any(p => line.IndexOf(p, StringComparison.OrdinalIgnoreCase) >= 0))
                    result.Add("=== Line " + lineIndex + ": " + line.Trim());

                line = stream.ReadLine();
                lineIndex++;
            }
        }
        catch (Exception e)
        {
            string ex = e.Message;
            Console.WriteLine("!!EXCEPTION: " + e.Message);
            Console.WriteLine("Continuing... ");
        }

        return result;
    }
}
}

The problem I'm having is when I try to get the projects using versionControl.GetAllTeamProjects(true); allProjs seems to get no information back. It is an empty array of TeamProject. When I debug, tfs.HasAuthenticated is true and it doesn't throw any exceptions. The TFS server is on a https domain. How can I make sure I'm exactly connecting to TFS?
Answer: I test at my side, everything works correctly.

Install the Nuget package
Microsoft.TeamFoundationServer.ExtendedClient by running below
command in Package Manager Console:
PM>  Install-Package Microsoft.TeamFoundationServer.ExtendedClient -Version 15.112.1
Make sure you specify the correct TFS server,
eg : http://servername:8080/tfs/DefaultCollection

Then check it again.
Comment: @BryanMcneil I investigate this but cannot find a way to search the Git as TFVC projects even with the GIT client API used (GetService()). If you are using TFS 2017, you can use Code Search feature.
Comment: Thanks for your answer. So I found out that our TFS is using Git, which is what I believe to be the problem. Do you have any ideas of what I need to change to get that to work?
Comment: Thanks for help. I'll look into adding the Code Search feature.


Keras : Should I care about the LAST epoch val_acc or only the HIGHEST epoch val_acc?

Question: I am running a grid search for a binary classifier. Each model will receive a list of parameters and then that model will be run 20 times (20 epochs.) Many times the first epoch will have a high val_acc and then the val_acc will decrease as the epochs move toward 20. Should I care that the val_accs are decreasing? Or can I just take the highest val_acc model and use that as the model?
Answer: The validation accuracy is perhaps the most important metric, in that it is a measure of how well your model has generalized, and how well it will perform on new unseen data.  If it is always decreasing your model is probably over fitting. Add more dropout layers, more/better training & validation data, or reevaluate your overall approach to the problem. To directly answer the question, yes you should care, your model is not learning. Good luck.


Why is indexing of UTF8 strings discouraged in Julia?

Question: The introductory guide to Julia, Learn Julia in Y Minutes, discourages users from indexing UTF8 strings:
# Some strings can be indexed like an array of characters
"This is a string"[1] # => 'T' # Julia indexes from 1
# However, this is will not work well for UTF8 strings,
# so iterating over strings is recommended (map, for loops, etc).

Why is iterating over such strings discouraged? What specifically about the structure of this alternate string type makes indexing error prone? Is this a Julia specific pitfall, or does this extend to all languages with UTF8 string support?
Comment: Since UTF-8 is a multi-byte encoding of Unicode, things like indexing into a UTF-8 string, or getting the length (in characters as opposed to code units) are O(n) (where n is the size of the string) operation. If you need to do things like that frequently, you'd be better off using UTF32String, and converting to/from UTF8 as needed.  This isn't a Julia specific pitfall, but some languages like Python 3 can pick the best internal representation for Unicode strings, that are still O(1) for indexing, depending on the string.
Comment: @ScottJones Thanks for the tip. UTF32String (which is similar to `std::wstring` in C++) is really what I need.
Answer: Because in UTF8 a character is not always encoded in a single byte.
Take for example the german language string böse (evil).
The bytes of this string in UTF8 encoding are:
0x62 0xC3 0xB6 0x73 0x65
b    ö         s    e

As you can see the umlaut ö requires 2 bytes.
Now if you directly index this UTF8 encoded string "böse"[4] will give you sand not e.
However, you can use the string as an iterable object in julia:
julia> for c in "böse"
           println(c)
       end
b
ö
s
e

And since you've asked, No, direct byte indexing issues with UTF8 strings are not specific to Julia.
Recommendation for further reading:
http://docs.julialang.org/en/release-0.4/manual/strings/#unicode-and-utf-8
Comment: @amrods you could use the `graphemes` function instead: `collect(graphemes("böse"))` returns `["b", "ö", "s", "e"]`
Comment: you can also iterate using `chr2ind` which converts from byte indices to character indices: `for i in 1:length(s); c = chr2ind(s, i); println(s[c]); end`.
Comment: Good point http://stackoverflow.com/users/2374329/ismael-venegas-castelló For a lot of use cases, using graphemes is really the best option, look at Swift 2.0's string handling, where graphemes are what you normally work with, not code points or code units.
Answer: Just to expand upon Scott Jones' comment, Julia actually also offers fixed-width strings similar to the std::wstring from C++, which allows for convenient indexing. They are now in https://github.com/JuliaStrings/LegacyStrings.jl One needs to install the package first with Pkg.add("LegacyStrings").
UTF32String would be the best choice for most use cases. To construct an UTF32String from a normal string: s2 = utf32(s).


Parse Integer From Text File Error With Java

Question: I am trying to write a function that is able to take a text file and separate each line into individual strings and adds them to an array, after which I will take the strings and convert the numbers into Integers or Doubles. However, it keeps returning a NumberFormatException whenever I try to do Integer.parseInt() on the first string in the array, which is always an integer.
This code is a simplified version of what I am attempting to do:
<code>import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

public class test {

    public static void main(String[] args) {

        try {

            File file = new File("preprocessed_data.txt");

            Scanner scanner = new Scanner(file);

            while(scanner.hasNextLine()) {

                String line = scanner.nextLine();

                if (line.length() != 0) {

                    // Splitting each line into an array of Strings
                    String[] strings = line.split("\\s+");

                    // Trying to convert the first String into an Integer
                    System.out.println(Integer.parseInt(strings[0]));

                }
            }

        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }

    }

}
</code>
This is the text file which I am trying to process:

11259   8111    +2    14   5.9   5.1  2.0 662.8449  1324.6825 1324.6817 0.0008   1     √     CDFEK($1)  KLTK($1)  [A1:5  P215:218]
10365   4551    +2    28  11.0   9.0  1.7 643.3196  1285.6320
  1285.6245 0.0075   1     √     CDFEK($1)  K($1)FR  [A1:5  P311:313]
16242   4175    +3    23  13.4   7.3  1.6 546.6142  1637.8280
  1637.8316 -0.0035   3     √     CDFEK($1)K  K($1)GDKAR  [A1:6  O448:453]
27030  24226    +3    16   5.4   6.4  1.7 893.4433  2678.3153
  2678.3178 -0.0024   2     √     KSFCAWLNVPNGNK($1)  IK($1)DNNMR + OxiM(22) 27031  25071    +3    10   4.8   5.1  2.6 893.4530  2678.3445
  2678.3178 0.0267   2     √     KSFCAWLNVPNGNK($1)  IK($1)DNNMR + OxiM(22) [A6:19  D503:509]
25104  18270    +3    19   6.8   5.8  1.7 805.7773  2415.3173
  2415.2965 0.0207   2     √     KSFCAWLNVPNGNK($1)  LRNLK($1)  [A6:19  I271:275  A6:19  I329:333  A6:19  I369:373]
27761  30048    +3    37   6.0   6.5  1.7 959.4729  2876.4041
  2876.3883 0.0158   1     √     KSFCAWLNVPNGNK($1)  ELNEQAGESK($1)  [A6:19  I469:478]
26769  27493    +3    17  13.0   6.4  1.3 883.4568  2648.3560
  2648.3541 0.0019   1     √     KSFCAWLNVPNGNK($1)  KPLDFEK($1) 26781  28982    +3    15   9.4   6.6  1.6 883.4586  2648.3611 2648.3541
  0.0070   1     √     KSFCAWLNVPNGNK($1)  KPLDFEK($1)  [A6:19  K1379:1385]

And this is the error which I keep getting:

Exception in thread "main" java.lang.NumberFormatException: For input string: "11259"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.parseInt(Integer.java:615)
    at com.company.test.main(test.java:25)
Comment: Does the debugger show any dodgy characters in the string you're trying to parse?
Comment: Your code works for me with exactly this file content. Seems like you have some characters at the begining of the line indeed.
Comment: It will be helpful if you add this line of code just prior to the Integer.parseInt() line;   IntStream.range(0, strings[0].length()).forEach(i -> System.out.println(strings[0].getBytes()[i]));;
This will print the byte value of each character, and might shed some insight to why the parsing is failing.  If any value is not between 48 and 57 you have a problem.
Comment: @IanMc Thanks, I tried that and it was indeed printing out numbers outside of that range. I figured out while trying to copy the text into a different file, that the problem was that the text file was being encoded as UTF-8 and when I changed it to ANSI when saving the file it worked. If you don't mind, could you explain why, when I put that code in, the value should be between 48 and 57, what does it mean if it is out of that range?
Comment: There are standard ASCII codes for all characters.
 The numbers 0-9 have ASCII codes 48-57.
 https://www.rapidtables.com/code/text/ascii-table.html
Answer: The text file I was attempting to process was encoded with UTF-8, and once I switched it to ANSI it removed the invisible characters in the beginning of the file and the code worked.
Answer: This looks like invisible characters, because your code works for me.
I would highly recommend opening the file in an editor like the notepad.exe on windows and make sure, that there are no invisible characters.


Drupal Webform REST

Question: I am trying to use Drupal Webform REST.
I got an error "The 'restful post webform_rest_submit' permission is required." on browser console. I have enabled modules and REST resources as mentioned.
I used Authorization, the Auth generated in Postman, using basic auth.
I am struggling to use 'x-csrf-token' in postman.
I want to use submit the form by an anonymous user. Do I still need Authorization, will just token not work on same-origin?
<code>const handleSubmit = async (e) => {
    e.preventDefault();

    await axios({
      method: 'GET',
      url: `${baseurl.URL}/session/token`,
      headers: {
        'Accept': 'application/vnd.api+json',
      }
    })
    .then(response =>  response)
    .then((token)=>{
      console.log("CSRF TODKEN", token.data);
      axios({
        method: 'post',
        url: `${baseurl.URL}/webform_rest/submit?_format=json`,
        headers:{
          'Accept': 'application/vnd.api+json',
          'Content-Type': 'application/json',
          'X-CSRF-Token': token.data,
          'Authorization':'Basic $S$EDSnVMXDICYbVGJ'
        },
        data: {
          "webform_id": "contact_form",
          "name":name,
          "email": email,
          "subject": subject,
          "message": message
        }
      })
    })
    .then(response => {
      console.log(response)
      response.status === 200 && console.log("Form successfully submitted")
    })
    .catch(err => console.log("SUBMIT FAIL ERROR ",err))```
</code>
Answer: It worked. I gave permission to "Access POST on Webform Submit resource" to anonymous users.

Drupal Webform REST


Do distortion pedals work well with tube amps? Or should I use a tube screamer?

Question: I have a Blackstar HT5 tube amp and I like to use a Boss DS1 with the clean channel for the heavy stuff (Zeppelin to Nirvana, these are very different I know).  Is this considered a good idea, or should I focus on getting my overdrive channel sounding right and use a Tube Screamer to enhance and boost the tone?  This is my first tube amp so this is all new to me.
Comment: Is a tube screamer not a kind of distortion pedal? Let's not forget the most important quote in tone-chasing: "If it sounds good, it is good".
Comment: This is an incredibly complex topic. I'll toss an answer down in a couple of hours or so.
Answer: In order to properly answer this question we need to identify that in the world of guitar effects there are three distinct ways to produce distortion (or clipping): overdrive, boost, and fuzz. Each has it's own unique characteristics. In addition, let's consider the three different ways to run a pedal + amplifier overdrive configuration: clean amp + pedal overdrive, overdriven amp + pedal overdrive, or finally just an overdriven amp. So given these relationships, let's talk about each and hopefully the answer to your question will shake out at the end. In essence, we are building what's called a tone stack.
First, let's quickly discuss the different types of overdrives. I'll tie these in later, so refer back here if you need to.
Fuzz: A fuzz circuit uses an electrical component called a transistor to produce clipping. The ubiquitous example of this is the legendary Dunlop fuzz face popularized by legends such as Jimi Hendrix, Pete Townshend, and David Gilmour. Although there are thousands of fuzz pedals on the market, the Fuzz face is the golden standard. Fuzz pedals tend to have their own personality especially if they are constructed with germanium transistors. They have a very wide range of tones, and blending them with an amplifier is a special case that I'll cover at the end.
Overdrive: Overdrive is a circuit designed to reproduce the distorted sound of a tube amplifier circuit. It does so via clipping which comes in two flavors: symmetrical and asymmetrical. Symmetrical clipping emphasizes odd order harmonics and consequently sounds harsh, while asymmetrical clipping mimics a push-pull amplifier and produces more even ordered harmonics. As a result, asymmetrical clipping will sound smoother, almost as if it were compressed. I go a little more in depth regarding clipping on this question.
Boost: Boost isn't actually distortion, but you use it to get more--note that the following information only applies if you have a tube amplifier or a boost placed directly in front of a JFET based overdrive pedal. Boost pedals come in many flavors, and you can actually use any pedal on your board that has a volume control that exceeds unity gain to achieve the same effect as a dedicated boost. Excluding uniquely built or small wattage push/pull designs a tube amplifier has two stages that we care about for our discussion: a preamplifier stage and a power amp stage. Each of these stages, when overdriven, is capable of producing distortion (via asymmetrical clipping). This is how a tube amp works. As you turn up the gain, it feeds more signal to the pre-amp and power amp stages causing the circuit to overdrive under load and produce distortion. So, couple this with the fact that a boost pedal is designed to take your guitar's signal and amplify it (ideally transparently) before it's fed into the preamp stage and what do you get? More distortion, and more volume. The volume 'goose' can be a problem for some sound engineers, but since boosts are typically utilized for solo applications the volume boost can actually be helpful in getting the solo to cut through the mix. That's another discussion though ;). You can also use boosts to feed more signal into an overdrive pedal--thus producing more clipping there as well.
Now. Armed with this information, let's talk about how to properly blend an amplifiers distortion with a pedal. Not all amplifiers like certain pedals. Darker amplifiers won't take to germainum based fuzz pedals unless a squishy muddy sound is your thing. Alternatively, brighter harsher amplifiers tend to dislike pedals with symmetrical clipping because it amplifies the harshness two fold. Typically there are three different ways to use pedals with an amplifier and each one has it's own set of problems. Let's discuss them.
Just Amplifier Distortion: Probably the simplest route to go is to not use any overdrive pedals at all. In this configuration you simply use the overdrive on your amplifier and augment it with modulation, compression, delay, and over effects. You could use a boost in this pattern to cut through for solos or overdrive your preamp stage past what the dimed gain control would do--which is very useful for getting more oomph out of an amplifier with less gain capabilities. Fender and similar amps come to mind here because they typically use preamp tubes with lower gain ratings such as the 12AU7. A boost could net you a little extra saturation of the preamp or power amp stage in this scenario. A potential drawback of this configuration is that it's locked in to the tone of your amp. Only do this if you are in love with your amplifiers stock tone, or play exceedingly high gain amplifier where an additional overdrive pedal will only add muddiness to the chaos. This pattern is the typical route for metal heads because it's simple, sounds good, and just works.
Clean amplifier pedal distortion: This configuration is where you turn the gain as low as possible on your amplifier and use pedals to get an overdriven sound. The benefit here is that if you have a single channel amplifier you can fake having a clean channel by simply turning off the pedal. Drawbacks of this are that you are stuck with the sound of the pedal you choose--and pedals never sound as warm and saturated as a real tube circuit. Tone models and JFETs can only do so much in the state that they are today. I usually don't recommend going this route unless you want to fake two channels, if for no other reason than you are defeating the fact that you have a tube amplifier. Buy solid state if you want to do this--they get much cleaner and are far better suited for this purpose.
Amplifier distortion and pedal distortion: The combinations are absolutely limitless here, but it takes lots of experimenting to get a tone that works. As we discussed earlier, tube amplifiers clip asymmetrically and emphasize even order harmonics. Pedals can clip either way--so the only way to figure out if something sounds good is to try it. Some general guidelines to follow though, are as follows. Note that YMMV depending on the type of music you play.

Get a low gain sound that you are happy with. This is called the golden "edge of breakup" tone, and will serve as the base layer for our tone stack. Think delta blues or SRV; something that sounds sweet and nice on open chords, but a little snarly when you dig in.
Next, add in your favorite distortion pedal at unity gain. Let me reemphasize that it should be at unity gain else you're boosting the channel and inducing more overdrive on the amplifier.
Now, tweak the settings of that pedal until it's where you want it. Do you want to use the pedal to completely saturate your tone? Turn up the gain on the pedal while reducing the volume until it satisfies you. If things begin to sound harsh or sterile, turn down the gain on the pedal and turn up the gain on the amplifier and that should warm your tone up. The idea is to find the optimal ratio of tube amp warmth to overdrive distortion and tone. The trickiest part is maintaining proportional volumes as you tweak settings. As you increase the gain on the distortion pedal it will drive more signal to the amp and thus overdrive the amplifier more--so you need to proportionally reduce the volume on the distortion as you increase it's gain rating.
Finally, add in any boosts you want to use and continue to tweak the blend until you have all your favorite tones nailed. One very important thing to remember about boosts is placement really matters. If you place a boost before an overdrive, then it will boost the input signal and cause the overdrive to clip more, but it won't do much to your amplifier's gain stages. Alternatively, if you place a boost after an overdrive or with no engaged overdrives in it's line of sight, then it will boost the signal to the amplifier and cause the preamp stage to distort more. Both of these are perfectly valid configurations so experiment with placement.

As you're going through this process, try to move the gain of the amplifier the least, because it serves as your base. Every time you change the base you'll likely have to completely reconfigure any pedals and boosts you have attached--so it's best to leave it as static as possible.
Finally, let's cover that special case. Fuzz pedals are awesome, but they are incredibly hard to blend with any amplifier or overdrive. Part of this comes from the magnitude and frequency ranges that a fuzz pedal tends to emphasize, and the tone of which will depend on the type and quality of transistors used to build the pedal. One of the major advantages of a fuzz, which should usually be placed very early if not first in your chain in order to sound good, is that they react to the volume control on your guitar much like a tube amp. As you turn down the volume, the fuzz pedal becomes much less saturated. You can use this trick to your advantage to make your fuzz play nice with the other overdrive producing components in your rig. By reducing the input volume to the fuzz pedal you can get a nice "edge of breakup" tone that we talked about in step 1 above without having to use your amplifier. Consequently, you can place your amplifier at a lower gain rating for a cleaner channel and also be able to blend your distortions and boosts with your fuzz. It doesn't work in every case for every amplifier, but that's par for the course. Using this strategy you could potentially get a nice saturated fuzzy tone by turning the volume on your guitar back to full with your entire tone stack (or pieces of it, whatever works best) engaged.
In closing, as with anything in the world of electric guitar gear you should always defer to the pirate codex: "They're not really rules. They're more like guidelines." What I write here might not work for you--but at a minimum you should be able to attack the problem analytically now. Make sure you write all your settings down, take a picture of them, or do something to document it. It's a shame to work for hours to get a tone where you want it only to forget where your settings were and lose it when someone dusts your amplifier + pedal board and moves the knobs on accident (personal experience). Now go experiment, and have fun with it!
Comment: Top drawer - I generally just twiddle knobs until it sounds good :).
Comment: This answer kicks ass.
Comment: Perfectly valid strategy too ;D.
Comment: I would think push-pull amplifiers would tend to have symmetric distortion, which would emphasize odd harmonics and be less grating than asymmetric distortion.  I think some of your statements are backward.  On the flip-side, push-pull amplifiers sometimes have a certain amount of crossover distortion which actually *decreases* with volume.  Crossover distortion *usually* makes quiet sounds quieter but more harmonically rich (if it doesn't mute them altogether) but in some amplifier designs it may make them louder instead, effectively imposing a minimum volume level.
Comment: I was using a clean Kustom single channel tube amp and distortion pedal combo for a while. It didn't sound great to me. I upgraded to an Orange TH-30, and I just tried the Boss Distortion and overdrive pedals on the clean channel, and it didn't sound as full and organic as the amp's built in distortion. I think those two particular pedals are great for solid state amps, but they don't leverage the amp to its full sonic potential.
Answer: It really depends what you want tone wise, I have a Blackstar HT studio in my home recording rig with a SD-1 in front, it sounds great, though I do tend to pile that onto the amps gain to get the tone I want.
A tube screamer(Bad Monkey or TS-9), would be equally good if slightly different (slightly smoother with an EQ on the Bad Monkey also).
There are a bunch of other overdrives you could try and I think you really need to try a few out and then decide.  I've been meaning to get a hold of the Blackstar Valve Drive pedal, purely because its just the best overdrive tone I have heard in all scenarios.  So you may want to try that one out.
As for Lead/Metal tones SD-1,TS-9,Valve Drive, will all do the job for you, and though the SD-1 is a little choppy (i still like it), they are very cheap.
The Idea is that you want to use the amps gain as much as possible to get the natural valve distortion, then just boost the tone a little with an overdrive/screamer of some sort.
All of these pedals produce a lot of noise (Valve drive less than the others), so to get the most out of them, you probably need to invest in a Noise Gate, i use an NS-2 on the floor.
Comment: @DRL, time will tell, I got mine only a couple weeks ago. My local music-shop got me one, so I can test it. The vendor was like wtf whe we un-packed and test-played it. (He never saw or heard one before, like me) Then we used a 'little' Marshall MR1960AV box (yeah, THE classic box) to test the external speaker jack... at this moment I knew that I need to take the amp, and get a V30 loaded cabinet. I also felt that 1 Watt CAN be overkill for some purposes... doesn't matter, I love it, my mom hates it. (she can't understand why I need one at home, when my band-room is so "near"... silly woman)
Comment: +1 I had a similiar question like Dre some weeks ago. And I can tell you are using wise words ;) In my home-equipment, I'm now using both, an overdrive and a distortion pedal in front of my HT1, to use them for different scenarios. I also plan to use my old NS2 to reduce the noise, once I get my ISP Decimator for my band-equipment and get a 2x12 with some V30 -> Ultimate bedroom rig!
Comment: That definitely answers my question.  I will be looking into either an SD-1 or TS-9 overdrive.  I will keep my DS-1 distortion in the mix but only to experiment with.  I like this idea as it is almost like having 3 channels, clean, overdrive (amp), and super (amp + pedal overdrives).  @Markus, I'm definitely curious as to what pedals you are using.
Comment: @Markus - The HT1 is a little animal of an amp for 1 Watt, best practice amp you could get for low volume IMO, and awesome tone for recording though the emulated output.  WOuld be interested to know how long the valves last
Comment: @Dre, I'm mostly using no-name pedals of the house-brand of my favourite shop, because they make awesome copies of well-known pedals. They're rock solid and often work as good/better the original, I compared some. (Don't ask me how they do this) For my Home-Equipment I got the Metal Muff copy http://www.thomann.de/de/harley_benton_extreme_metal.htm and the overdrive is this one: http://images.thomann.de/pics/prod/256803.jpg (rather hot OD) I also got a Boss MT-2 copy, a digital delay and the orig. Boss NS-2. The 2x12 is of the same brand, loaded with Celestion V30 for ~200 bucks, it's crazy.
Comment: Just a follow up, I ended up buying the SD-1.  Wow.  I'm using it as a boost and it is perfect for enhancing leads, or when I want some extra oomph to the existing gain.  I even dialed back the amp gain a little as going to 10 felt to be a bit much, but that is my personal preference. With just the SD-1 as boost and no amp overdrive it also sounds nice for a little crunch. I will keep using the DS-1 distortion on clean to mix things up, but this new setup is my preferred tone now.  Thanks again for the great advice.
Comment: No probs, sounds like a similar setup to mine, the SD-1 is truly a bargin :)
Answer: Firstly, read jduv's response. It is comprehensive and provides a view of the options.
Your final choices will be subject to your ear and whether you are gigging or recording.
For recording,  test all the above techniques and try to find other musos that will swap pedals so you have access to many different tones.
For gigging, I tend to use a limited set of basic overdive sounds and one very easy way to give yourself a wide set of tone and gain options is to use a EQ (such as GE7) in front of a good valve amp pre-amp. This allows gain boost for solos/heavey disto but also allows you to shape the tone so you can scoop or add mids etc .. (It wil also teach you what you like so you can look for pedals with these characteristics).
If you set the crunch channel on a mid to heavy setting you can clean up the sound by backing off the guitar volume. This affects gain to the pre-amp, not the power amp volume. I also use a volume pedal after the pre-amp so I can add more volume for cleanish lead or to regain lost power when my guitar volume is dropped a lot. It also allows you to add high gain (with a pedal?)to the pre-amp but back off the output volume. You can also get very smooth violin affects with the volume pedal here. There is no reason that you can't have a distortion pedal in this set up, but unless it gives you a specific sound, you don't need it.
Comment: +1 Very cool! I never thought of putting volume there (it would seem there's enough volume controls in that part of the chain already).


Find the argument form for the argument and determine whether it is valid

Question: I have been trying this question for about 3 hours and I am still stuck please help.
Q: Find the argument form for the argument and determine whether it is valid. Explain your reasoning.

If I get a bonus, I will buy the painting.
If I sell my books, I will buy the painting.
∴ If I get a bonus or sell my books, I will buy the painting.

We have to build and solve this argument using the rules of inference, however, I have not found a way to do it so far.
What I have done:
List of premises: p --> q, r --> q,
Conclusion: (p V r ) ---> q

p --> q (if p then q )
r --> q
 3.

The question should be solved like this example:
Ex: Show that the premises "It is not sunny this afternoon and it is colder than yesterday," "We will go swimming only if it is sunny," "If we do not go swimming, then we will take a canoe trip," and "If we take a canoe trip, then we will be home by sunset" lead to the conclusion "We will be home by sunset."
Solution:

¬p ^ q   (Premise)
¬p        (Simplification using 1)
r --> p (premise)
¬r   (Modus tollens using 2 and 3)
¬r --> s (Premise)
s         (Modus ponens using 4 and 5)
s --> t  (Premise)
t          (Modus ponens using 6 and 7)

List of available inferences:
modus ponens
modus tollens
hypothetical syllogism
disjunctive syllogism
addition
simplification
conjunction
resolution
We can also use logical equivalences including the ones with conditional statements
Comment: Sometimes validity is easier shown in a truth table. All you need to do is show that whenever you have $p \to q$ true and $r \to q$ true, you always have $(p \lor r) \to q$ true
Comment: Your example shows *some* of the rules that are apparently available for you to use, but what *other* rules do you have?
Comment: Check the updated question at the very end for list of inferences
Comment: Youre right about the truth table but the question asks to present it in an argument form such as the example above
Comment: How is the resolution rule defined?
Comment: If you have two premises in the form of p v q and ¬p v r then the conclusion will be qvr
Comment: Or as a tautology we can do: ((pvq) ^ (¬pvr)) --> ( q v r)
Comment: We can also use logical equivalences including the ones with conditional statements
Answer: $\def\fitch#1#2{\quad\begin{array}{|l} #1\\\hline #2\end{array}}$
To use a conditional proof: assume the antecedent aiming to derive the consequent, then (if successful) apply the rule of "Conditional Introduction".
$$\fitch{\text{Premises}}{\fitch{\text{Antecedent}}{\vdots\\\text{Consequent}}\\\text{Antecedent}\to\text{Consequent}}$$
So in this case, you wish to show $p\to q, r\to q \vdash (p\vee r)\to q$.   As the assumption you need to make is a disjunction, you need so rule of inference that will eliminate a disjunction...aka a proof by cases argument.
$$\fitch{1. ~p\to q\\2.~r\to q}{\fitch{3.~p\vee r}{\vdots\\X.~q\qquad\qquad\text{... disjunction elimination}}\\Y.~(p\vee r)\to q\quad\text{3-X,conditional introduction}}$$


Sampling Methods for Review of Many Hours of Surveillance Video Footage?

Question: I'm curious is anyone has experience with or can point me to some resources for sampling methodology for reviewing surveillance video footage. For example, say we have 6480 hours of video footage (3 months by 30 days x 24 hours x 3 cameras) and we are interested in characterizing some activity that occurs (not necessarily enumerating). Say we know that individuals that engage in this activity tends to be present at night and more likely to be present during second month than the first or third months. The activity may happen multiple times throughout the monitoring period, so it isn't a matter of capturing it just once, nor are we interested in capturing every instance of the activity. We just want to sample enough of it to be able to characterize the population engaging in this activity. So by way of example, say the activity occurs 1000 times during the monitoring period (being more prevalent at night and during the middle of the monitoring period) and takes between 10 and 30 minutes per instance. The activity can overlap i.e. several individuals may, but don't necessarily have to, be engaged in the activity at the same time on screen. How do we sample this footage (say selecting 10 to 20 percent of the total footage) to be able to maximize the number of instances we observe while not wasting labor reviewing all the footage?
EDIT: As has been alluded to in some of the comments, there are actually two things going on here. The first is that I want to sample footage in such a way that I maximize the possibility of an individual being present on screen. Given that a person is present they may be engage in some activity, what I want to do is characterize the range of behaviors. For this I want to be able see enough individuals that I can say that I've seen representative selection of their behaviors. I have data on the first part of the problem. Without revealing too much, let me give you that actual data to play with. What follows are total counts of individuals seen on footage from a camera nearby the location I am interested in. I have summarized these counts first by hour and secondly by day of year. For example over five years, 1005 individuals were seen in hour 0 (midnight to  1:00 AM) during the three month monitoring window. Similarly, over five years, 29 individuals were seen on day 182 (at any hour). My idea is to use these counts to come up with weights for a weighted random sample. As a side note, if I were interested in estimating the total number of individuals that I would see if I viewed all of the footage I do realize I can use a Horvitz-Thompson estimator.
<code>tab.24<-data.frame(hour<-seq(0,23, 1),
count<-c(1005, 851, 750, 562, 311, 176, 132,  83,  99,  93,  87,  83,
89,  82,  83, 114, 187, 148, 152, 199, 398, 767, 1002, 1100))
tab.day<-data.frame(doy<-seq(182,273),
count<-c(29, 19, 18, 31, 25, 24, 44, 49, 54, 42, 38, 75, 71, 62, 71, 96,
142, 157, 117, 77, 159, 107, 104, 126, 134, 91, 146, 192, 223, 181, 201,
210, 178, 128, 203, 204, 250, 176, 243, 196, 213, 187, 200, 167, 160, 141,
120, 162, 178, 171, 145, 103, 99, 75, 114, 148, 140, 81, 69, 77, 49, 59, 44,
54, 58, 51, 46, 47, 54, 49, 35, 35, 39, 46, 46, 60, 20, 22, 37, 18, 22, 28,
38, 29, 30, 17, 22, 10, 18, 9, 15,3))
</code>
Answer: First let's assume assume you truly want to maximize incidence of this activity in your sample.  Let's also assume that the smallest clip we're willing to sample is fixed, let's say 1 hour for simplicity.  In order to frame this in some probabilistic theory, it would be nice to be able to assume independence between the number of people doing this activity in any interval, which of course is not reasonable since if someone starts in the last couple minutes of one hour, they'll be doing it in the next.  So, if you don't mind, I will approximate the problem by saying we want to capture as many people starting this activity as possible.  Beside this, is the independence assumption reasonable in your case?
Moving on, with this set of assumptions, our only choice is, which hour of footage to watch next.  Therefore the only relationship we can exploit is the relationship between time and the rate of this activity.  So if we model this rate as a function of the hour, than hour of day, or day of week, or is a holiday, etc. are our only real features (unless you can extract features from the video like average brightness...)
For continuing this problem I would look into two things.  First, The Poisson distribution which is the distribution used for modeling rates of independent events.  Secondly, the multi armed bandit problem.  Essentially, think of your footage as a slot machine, and each hour of day as a lever.  The payoff is instances captured.  When you sit down at the slot machine you can start with some beliefs (as it sounds like you do) in the form of a (probably gamma) prior of which lever is best, or no beliefs at all.  But each time you play, you update your beliefs, informing your next draw.
Answer: Very interesting to think about sampling to solve such a problem ! I'd suggest stratified sampling, using 4 stratas :

Day, 1st and 3rd month
Day, 2nd month
Night, 1st and 3rd month
Night, 2nd month

About the allocations within each stratum, do you have an (even rough) idea of how many such activities occur within each stratum ? If you do, you can use the proportional allocation (which gives optimal precision on estimators withing each strata). For example, if you know that when you observe 100 activities, they divide up like this :

5
10
35
50

Then, if you want to sample 100 hours out of 4000, you can draw 4 simple random samples in each stratum like this :

5 hours (out of 1000)
10 hours (out of 1000)
35 hours (out of 1000)
50 hours (out of 1000)

Of course, you can define your strata in many other ways (or use more strata it matches the parameters of your problem better), but that's the general idea of stratified sampling.
EDIT : I did not mention unequal probability sampling on purpose. You could very well use probabilities proportional to the number of activities observed (say, per hour per day). This would give you an Horvitz-Thompson estimator of the total number of activities with minimum variance (0, in fact, if your probabilities are truly built upon the real number of observed activities).
However, if you try to estimate an Horvitz-Thompson of a variable that is weakly correlated with the number of activities, you could end up with a very bad precision. In your case, I understand you try to estimate a discrete variable describing characteristics of people engaging in the activity, and I take it you don't have prior knowledge on how these variables behave ?
This is why I proposed stratified sampling : it will perform reasonably well on the estimate of the total number of actitivies (and using the proportional allocation, you will observe a not-far-from maximum number of activities), and you don't risk ending up with very high variances on other estimators.
Comment: Antoine.  Stratified sampling is an important technique for trying to get a representative sample of a population.  Bug the OPs question regards maximizing the number of incidents captured.  This is the goal in his sampling.  What does strata fixation achieve there?
Comment: The way I understand the question, the goal seems to be to achieve minimum variance on various estimators of characteristics regarding people engaging in said activties, not truly maximize the number of activities observed in sampled footage. If the strata are wisely chosen, I think the total number of activities observed will not be too far from maximum given the sample size.
Comment: I guess both of these objectives are explicitly stated, and they are really at odds with each other, if we assume that frequency and population characteristic correlate with time (in a non orthogonal way).
Comment: I've been pursuing the idea of unequal probability sampling, but not necessarily stratified sampling. I'll edit my post shortly with some actual data and a clarification of the question.
Comment: Unequal probability sampling is seldom used in my field (official household stats), mainly because it is "risky" (in terms of precision). See my edit.


How to quickly obtain PHP localised ordinals using Zend_Locale

Question: I need to localise ordinal values i.e 1st, 2nd, 3rd etc.. into various european languages There doesn't seem to be an option for doing this with Zend_Locale. Anyone know of a way of doing this in 5.2.x without the intl pecl extension?
To be clear, 5.3 is not an option and installing an additional extension may not be either. Anyone know of a way of doing this using Zend_Locale?
thanks!
codecowboy
Comment: your best option would probably be Zend_Translate using the array adapter
Comment: I agree I've been hunting around for a solution here and kept coming back to Zend_Translate.
Comment: I also think that is the best solution, since it's not supported by zend_locale as you mentioned before. Actually I also think this is rather a "translation" issue instead of a localization issue...
Answer: Another option is to connect to Google translate using Zend_GData, more info on the API; http://code.google.com/intl/nl/apis/gtt/
Don't forget to save or cache the responses requested.


help for showing this Inequality

Question: $x\in l_\infty$
Definition:
$$p(x):=\limsup_{i\to\infty}x_i:=\inf_i(\sup_{j\geq i }x_k)$$
show that $p$ is sublinear. In other words:
1.$p(x+y)\leq p(x)+p(y)\quad\forall x,y\in l_\infty$
2.$p(\alpha x)=\alpha p(x)\quad\forall x\in l_\infty\quad\forall \alpha\in\mathbb{K}$
Comment: Since your in the space of bounded sequences the limit should always exist. 1 should then follow from the triangle inequality on numbers. 2 should follow from the usual properties of limits.
Comment: Number 2. is only required to hold for $\alpha \geqslant 0$. And it does not hold for $\alpha < 0$ in general. (And $\limsup$ only makes sense for real sequences, not complex, so $p$ is only defined on the real space.)
Comment: Tank you for the points
Comment: Please check if my arguments below in the answer is correct.
Answer: We have to prove $$p(x):l_{\infty}\to\mathbb{R}$$ is a sub-linear mapping. We $p(x)$ is denifed as
$$p(x):=\limsup_{i\to\infty} x_i$$
$x=(x_i)_{i=1}^\infty\in l_{\infty}$
I am going to use this definition of
$$\limsup_{i\to\infty} x_i:= \lim_{i\to\infty}\sup\{x_j :j\geq i \}$$

$p(x+y)\leq p(x)+ p(y)$

$$p(x+y):=\limsup_{i\to\infty} (x_i+y_i):= \lim_{i\to\infty}\sup\{x_j+y_j :j\geq i \}$$
using supreme of a sum is lower or equal than sum of the supremes, we have
  $$\lim_{i\to\infty}\sup\{x_j+y_j :j\geq i \}\leq \lim_{i\to\infty}(\sup\{x_j :j\geq i \}+\sup\{y_j:j\geq i\})=p(x)+p(y)$$
since we can split the limit of a sum into a sum of limits.

$\alpha p(x)=p(\alpha x)$ $\quad\forall \alpha\geq 0$

$$p(\alpha x):=\limsup_{i\to\infty} \alpha x_i:= \lim_{i\to\infty}\sup\{\alpha x_j :j\geq i \}$$
$\alpha$ will "get out" of $\sup$ since $\alpha\geq 0$  and "get out" of limit since constants can do it in limits. So  $\alpha p(x)=p(\alpha x)$ .

help for showing this Inequality


removing a trailing slash from a filename (index.php/ to index.php)

Question: I'm sure I'm missing a trick here but I can't work out how to remove the trailing slash from a file name in a URL using an Apache RewriteRule.
My URL would be something like this:
www.mysite.com/dynamic_folder_name/index.php/
I need the URL to become
www.mysite.com/dynamic_folder_name/index.php
Any help would be appreciated.
Cheers
Comment: edited your question for clarity.
Answer: Where do you want to do this?
In an Apache rewrite rule?
In a PHP page?
In a bash script?
Plase specify...

Update
RewriteRule. Ok. This should do what you need:
RewriteRule ^(.+)/$  /$1 [R=301,L]

Comment: One more example of why people should have to leave a comment when down-voting (even if that comment is anonymous).
Comment: @Massimo the RewriteRule only says to remove the slash, but if the DirectorySlash directive is set to "ON", which is the default, it would cause them to each override the other (ie. a loop). See http://httpd.apache.org/docs/2.0/mod/mod_dir.html#directoryslash
Comment: Sorry, as an apache rewrite rule, so a .htaccess file
Comment: ...why the downvote?!?
Comment: hmm wasn't me :/
Comment: Because people are overzealous.  I canceled it out.
Comment: Thanks Massimo, that works perfectly...so simple.  Can't believe I missed it!
Comment: @eek, regular expressions *are* quite tricky...
Comment: Won't this rule cause a redirect loop if you try to access any directories? For example www.mysite.com/dynamic_folder_name
It might be better to modify it to look specifically for the php extension or something similar before redirecting the trailing slash.
Comment: @matthew: no, it only removes a *trailing* slash, thus there can't be another one after having removed it; besides, the "L" flag tells pache "don't process any other rewrite rule after this one".


How to derive this binomial identity?

Question: I believe the following is an identity (I've tested with a few random $m$ and $n$ values, could be wrong though):
$$\sum_{k= 0}^{\infty}{m \choose k}{n \choose k}k=n\binom{m+n-1}{m-1}$$
but I'm not sure how to prove it.
Initially I thought of considering the right side combinatorially: pick $m-1$ things from a bucket of $m+n-1$ things and repeat this $n$ times. But how do I think of the left side combinatorially, since there is an infinite sum? Or maybe this is the incorrect approach?
Comment: Start by noting that $\binom{n}{k}k=n\binom{n-1}{k-1}$.
Comment: According to [Wolfram Alpha](http://www.wolframalpha.com/input/?i=summation+from+k%3D0+to+infinity+of+(m+choose+k)(n+choose+k)k), you're right. (Note that $\Gamma(x)=(x-1)!$ for $x \in \Bbb{N}$.)
Comment: This is **essentially** equivalent to https://math.stackexchange.com/questions/2159846/please-help-me-with-this-combination-problem , but there is some fidgeting required to convince yourself of the equivalence (the bounds of the sums are immaterial, and the case $m+n=0$ needs to be checked separately).
Answer: Hint
Note that $k\binom{n}{k}=n\binom{n-1}{k-1}$, and erase $n$ from both sides.
There is one box with $m$ balls, and the other with $n-1$ balls.
You are trying to pick $n$ balls in total.
Note that this is equal to $(RHS)$.

 Note that $\binom{n-1}{k-1}=\binom{n-1}{n-k}$
 Since you are trying to pick $n$ balls in total if you pick $k$ balls from the first box, you are going to have to pick $n-k$ from the second. This operation is equal to $(LHS)$.
Comment: Okay so we can show $\sum_{k=0}^{\infty}\binom{m}{k} \binom{n-1}{m-1}=\binom{m+n-1}{m-1}$.  But I'm not sure where to go from here. How to handle infinite sum?
Comment: @User I am using that $\binom{m+n-1}{m-1}=\binom{m+n-1}{n}$. Also, note that if $\binom{n}{k}$ would become $0$ if $k > n$, so the infinite sum does not matter as much.
Comment: I see it now. That trick to remove the infinite sum is what I was stuck on.
Answer: $$\begin{align}
\sum_{k= 0}^{\infty}\color{green}{{m \choose k}}\color{blue}{{n \choose k}k}
&=\color{blue}n\sum_{k=0}^\infty\color{green}{\binom m{m-k}}\color{blue}{\binom {n-1}{k-1}}\\
&=n\binom{m+n-1}{m-1}\qquad\blacksquare\end{align}$$
using the Vandermonde identity. Applicable limits for $k$ are $1\le k\le \min(m,n)$ as $\binom ab=0$ for $a<b$.
Comment: I was unaware of this identity. It looks very useful. Thank you.
Comment: It is indeed! You're welcome.
Answer: Since you were originally aiming for a combinatorial proof, I thought I'd offer one: a committee consists of $m$ members from the minority party and $n$ members from the majority party.  A subcommittee is to be formed having an equal, but unspecified, number of members from each party, and one of the subcommittee members from the majority party is to be designated chair of the subcommittee.  Call the result a marked subcommittee.
The left side clearly enumerates the number of marked subcommittees.  To see that the right side does too, observe that the majority-party and minority-party members of a marked subcommittee may be paired up as follows: line up the subcommittee members from the majority party with the chair first and everybody else in alphabetical order.  Line up the subcommittee members from the minority party in alphabetical order.  Now pair up the first people in each of the lineups, the second people in each of the lineups, and so on.
We can now interpret the right side of the identity: select a person from the majority party to be subcommittee chair.  Now line up the $n$ majority-party members with the chair first and everybody else in alphabetical order.  There $n$ possible lineups.  Place $m-1$ separator marks in this lineup to partition the lineup into $m$ parts (some possibly empty).  By stars-and-bars, there are $n\binom{n+m-1}{m-1}$ partitioned lineups.  Line up the minority-party members in alphabetical order and pair up the first minority-party member with the first part of the majority-party lineup, the second minority-party member with the second part of the majority-party lineup, and so on.  Then the subcommittee is formed by taking those minority-party members that have been paired with a non-empty part of the majority-party lineup, and by taking as majority-party members the lead member of each such non-empty part.  Note that since the chair will be the lead member of her/his part, the chair will always be taken.
Example: To show how the correspondence works, let the majority party consist of $A$, $B$, $C$, $D$, $E$ and the minority party of $1$, $2$, $3$, $4$.  Suppose the subcommittee consists of $B$, $D$, $2$, $3$, with $D$ as chair.  Then the pairing between minority-party and majority-party subcommittee members is $(2,D)$, $(3,B)$.  The corresponding partition of the majority-party lineup is $\lvert DA\lvert BCE\lvert$, the associated pairing is $(1,())$, $(2,(D,A))$, $(3,(B,C,E))$, $(4,())$.
Going the other way, suppose $B$ is chosen as chair and the partition of the majority-party lineup is $B\lvert\lvert ACD\lvert E$.  The associated pairing is $(1,(B))$, $(2,())$, $(3,(A,C,D))$, $(4,(E))$, which produces the subcommittee consisting of $B$, $A$, $E$, $1$, $3$, $4$, with $B$ as chair.
Comment: An alternative explanation of how the right side counts marked subcommittees: from the $n$ majority party members choose a chair. Then choose $m-1$ of the remaining people (of both parties). Of the chosen people, any from the majority party are placed on the subcommittee; any from the minority party are LEFT OFF of the subcommittee.  If $k-1$ of the chosen people are of the majority party, then, with chair included, the majority party will have $k$ members on the subcommittee.  The minority party will have $m-1-(k-1)=m-k$ members NOT on the subcommittee, and hence $k$ members ON it.

How to derive this binomial identity?


Bootstrap 4 modal won't close

Question: I have a Bootstrap 4 modal that pops up on button click but will never close (except on refresh). I'm trying to make it close by clicking the 'close' button or by clicking on the main page body.
I've tried every combination of the class, data-toggle, target, etc. attributes from this page: https://getbootstrap.com/docs/4.0/components/modal/
Here's the code to open the modal:

            Create Event

Here's the code for the modal itself:

Create a new event

              <button type="button" class="btn btn-default" aria-label="Close" data-toggle="modal" data-target="#createModal" class="close" data-dismiss="modal">Close</button>

Parr Family

                                  <input type="text" class="form-control" placeholder="Add event" value="" />

                                  <input type="text" class="form-control" placeholder="Description" value="" />

                                      <input type="radio" name="person-select" value="helen" checked>
                                       Helen

                                      <input type="radio" name="person-select" value="bob">
                                       Bob

                                      <input type="radio" name="person-select" value="violet">
                                       Violet

                                      <input type="radio" name="person-select" value="dash">
                                       Dash

                                      <input type="radio" name="person-select" value="jackjack">
                                       Jack-Jack

                                  <input type="date" class="form-control" placeholder="Date" value="" />

                                  <input type="time" class="form-control" placeholder="Time" value="" />

                              <input type="submit" class="btnRegister"  value="Create"/>

I'm fine with either method of closing it, I just need it to close in some way.
Edit: Adding 'modal-dialog' as a class to the second div allows me to close the modal, but it also adds a lot of unwanted styling.
Second edit: I moved 'modal-dialog' to the next div and it fixed everything.
Answer:

var modal = document.getElementById('createModal');
function Close() {
    modal.style.display = 'none'
}
var close = document.getElementsByClassName('close');
close.addEventListener('click', Close);
Answer: Maybe you have more than one modal in the page.You can give an id to the close button and use jquery to close the modal. Let's say the id="closeBtn". Then, you can close the modal using
$('#closeBtn').click(function(e){
    e.preventDefault();
    $('#createModal').modal('hide');
})

Comment: Sorry, I do have more than one modal! I gave this one a specific ID, as well as its button, and followed your suggestion and it still doesn't work. Thank you!

Bootstrap 4 modal won't close


Printf'ing floating point numbers in C++ gives zeroes

Question: I am doing some c++ right now and stumbled on a problem I can't get to wrap my head around.
I am doing a floating point comparison like this:
if(dists.at<float>(i,0) <= 0.80 * dists.at<float>(i,1)) {
  matched = true;
  matches++;
} else {
  printf((dists.at<float>(i,0) <= 0.80 * dists.at<float>(i,1)) ? "true\n" : "false");
  printf("threw one away because of dist: %f/%f\n", dists.at<float>(i,0),dists.at<float>(i,1));
}

at the first line, the comparison threw a false what means: dists[0] > dists[1]
When we print the values, the results are this:
falsethrew one away because of dist: 0.000000/0.000000
falsethrew one away because of dist: 0.000000/0.000000

I think it has something to do with the results not being a float or something, but I'm no pro at C++ so I could use some help to find out what these values are.
Comment: Not enough digits printed? `float` provides up to 7 digits of precision.
Comment: "7 digits of precision" doesn't mean "to 0.0000001", it means having 7 significant figures (in decimal). `FLOAT_MIN` is something around 1E-37.
Comment: Set a breakpoint, assign the float to a variable (`auto tmp = dists.at(i,0);`), and see if `tmp` is really 0, or even really a float, in your debugger.
Comment: Nup, dist: %1.7f/%1.7f\n gives the same results:
Comment: %g instead of %f can print very small floating point numbers if it is the problem.
Comment: Or `%e` for scientific notation.
Comment: Ahh that was the problem, it was indeed a very very small number hehe. thanks for the help. %g worked
Comment: @Keugyeol: kindly post your comment as answer for the community
Comment: What does `dists.at` evaluate to? What type?
Answer: Looks like it was due to the number being too small.
In response to @NirMH, I am posting my comment as an answer.
%g instead of %f can print very small floating point numbers.
There are other options such as %e, as @ilent2 mentioned in the comment.
For example:
double num = 1.0e-23;
printf("%e, %f, %g", num, num, num);

prints out the following result:
1.000000e-023, 0.000000, 1e-023


Interplay of Categorical Data

Question: I apologize if my phrasing is un-educated, I don't have much of a stats background. I'm struggling to find info answering my question because I lack the terminology to accurately describe it.
Let's say that I was reviewing data on mp3 player sales, and wanted to see if certain combinations of categorical data resulted in returns.
Things like color, memory size, brand. I don't simply want to see the relationship between returns and the color red, but rather which combinations indicate a likely return.
For example, imagine that a certain manufacturer uses a different factory to produce each color mp3 player - and the factory that produces blue ones has a defect that shows up 3 months later and is causing returns. I'd be able to see a relationship between that color and brand with returns in that example. Does that make sense?
What test could I leverage for this? My jumping off point is a chi-square, except that I can't see the interplay between the different variables with that. I haven't been able to figure out where to go from there.
Answer: I recommend first getting a good display of the results, so that you can be oriented. To that end, I think a mosaic plot is quite effective. You take a rectangle and divide it into panels in proportion to one variable. Then subdivide each panel in proportion to the frequencies of a second variable within the first one; and so forth. Here is an example with data on survival on the Titanic by sex, class, and age. (In this diagram, green is "survived" and blue is "died".)
Comment: This looks good. I'm a little concerned with exploring the frequency, though, instead of the return rate - the results could be drowned out by higher selling models in my example. For instance, perhaps I only sell one of those defective blue ones for every 100 red ones I sell - but let's say 90% of the blue ones are returned. How can I better explore that ratio?
Comment: Yes. In regression, including logistic, the predictors can be either factors or measurements (but you need to designate the factors as class variables or make indicators out of them). It'd be more convenient to tabulate the frequencies of 0s and 1s, rather than having zillions of observations, one for each customer. That's where software differs on how you set it up. Some use counts of successes and failures, some use successes and totals, etc., so you have to make sure you're giving it the right information.
Comment: Not sure what you mean - is it that there are some cases that don't occur in the data? Those *do* cause some concern in the asymptotics of the tests, just like in a chi-square situation. You kind of have to grit your teeth and don't get too literal in interpreting the results.
Comment: They might still be visible - note the small proportion of children in my example (but also that children of crew members are NOT visible). For actual analysis, you probably should use logistic regression. Are you familiar with that? Unfortunately, software varies quite a bit on what you need to do to set it up. We can try to help though.
Comment: Can I use logistic regression with nominal data? I've looked into it a bit - would my approach be to seperate each color and brand as a seperate binary variable? Red 1/0, Blue 1/0, Green 1/0, Brand_1 1/0, etc.?
Comment: This sounds like what I need, but just to be overly clear, let me ask you this - there's no cause for concern that one mp3 player will be a 1 for a color, a brand, and a memory size, but never a 1 for more than one color, more than one brand, etc.?
Answer: Decision trees are a good classification and feature selection model for categorical data. They are very simple to understand and provide a good view of the relationship of attributes as you can see in the image. Each node provides a test in a selected attribute, and attributes can be aggregated. I believe this is a good starting point for you.

Interplay of Categorical Data


Not a function despite passing vertical line test?

Question: I'm confused with something once again in my notes. The question is,

$R$ on $\mathbb R$, $R = \{ (x, y): y = \sqrt{x} \}$ is not a function

how is the graph $y = \sqrt{x}$ not a function?

$ y = \sqrt{x} $
$ x = y^2 $
Dom of $R = \{ x \in \mathbb R : x \ge 0 \} $

This is what the graph should be accordingly to the condition above

Then i got another question that says

R on $\mathbb R$, $ R = \{ (x, y): x = y^2 \} $ is a function

which isn't it the same as the above?
Can anyone kindly explain?
Comment: $$\Bbb R_{\ge 0} \to \Bbb R \\ x\mapsto \sqrt{x}$$ *is* a function.
Comment: @TeoChuenWeiBryan The key part is where it says "on $\Bbb R$".  Notice all of us said it's a function on $[0,\infty) = \Bbb R_{\ge 0}$.
Comment: Of course it is a function on domain $[0,inf)$ It is even a one-to-one function!
Comment: It's a function on the domain $[0, \infty)$
Comment: But according to the notes it says that `{(x, y): y = √x} is not a function`
Comment: To recap everything said... The second one, assuming you are intending $\{(x,y)~:~x=y^2\}$ includes points $(4,-2)$ as well as $(4,2)$ since $4=(-2)^2=(2)^2$ and so it fails to be well defined (some input has multiple outputs).  The first is a function over domain $\Bbb R_{\geq 0}$ but is not a function over domain $\Bbb R$ as it is not everywhere defined (some inputs have no outputs).
Comment: For the first question, i do understand why it is not a function because it does not satisfy the condition of a ℝ. However, it is the same for the 2nd which is x = y square. Unfortunately, they considered that as a function even though they have the same domain.
Comment: Are you sure the second wasn't supposed to be $y=x^2$?  That is a function on $\Bbb R$.
Comment: The second would be considered a function if both domain and range is equal to $\Bbb R_{\geq 0}$
Comment: No. That is why i'm so confused
Comment: By the way that *isn't* what the graph should look like.  That's the inverse.  The graph should look like y = x^2; x \ge 0.
Answer: The question you include seems to be asking:

Is $\{(x,y):y=\sqrt{x}\}$ a function $\mathbb R\rightarrow \mathbb R$?

You are correct both in that $\{(x,y):y=\sqrt{x}\}$ is a function and that the domain of this function is $\{x\in \mathbb R:x\geq 0\}$. However, the answer is therefore "No" because the domain of the function is not $\mathbb R$, as would be required of a function $\mathbb R\rightarrow\mathbb R$.
If you are given the domain, then when you apply the vertical line test, you require that, for each $x$ in the domain there is exactly one $y$ such that $(x,y)$ is in the function. Here, the function fails, for instance taking $x=-1$ where no pairs $(-1,y)$ are included in the given relation.

For the second question, note that $x=y^2$ has two solutions for $y$ for positive $x$, which are $\sqrt{y}$ and $-\sqrt{y}$. Thus, if you choose any positive $x$, you can relatively easily find that two pairs are included of the form $(x,y)$, so it fails the vertical line test. Try plotting the set of pairs like you did with the square root function.
Comment: This makes sense, however if the graph provided by the OP was the same graph the OP had to do on a test, then this is a TERRIBLE worded question, if the answer would have been "no function"
Comment: I do get why it is not a function because it fails to satisfy the condition as a real number as x is only more than or equals to 0 which does not include the negative number. However, as for x = y square, the domain does not include all the values in a real number too. Unfortunately, that is counted as a function which is like some sort of a contradiction.
Comment: @TeoChuenWeiBryan I added something to the second part of your question. The second thing is definitely not a function - it both has places where a vertical line intersects the graph twice and where it intersects the graph zero times, either of which disqualifies it from being a function with domain $\mathbb R$.
Answer: A relation being considered a function has as much to do with the domain and codomain as it does the relationship itself.
A visual interpretation of the definition of a function, $f$, with domain $X$ and codomain $Y$ is as a directed graph with a vertex representing each point in the domain on the left, a vertex representing each point in the codomain on the right, and a directed edge between a point on the left to a point on the right if they are related.

For example, with domain $\{a,b,c\}$ and codomain $\{\text{apple},\text{banana},\text{cherry}\}$ and relation a letter is related to a word if the letter begins that word.
We have the image:

To be a function: EVERY vertex from the domain must have exactly one arrow leaving it going to the codomain.  There cannot be zero arrows and there cannot be more than one arrow.

The above example satisfies what it means to be a function then.

If we change the example somewhat, and have domain $\{a,b,c,d\}$, codomain $\{\text{apple, banana, cherry}\}$ and the same relation (a letter is related to a word if the letter begins that word)

This is no longer a function because the letter $d$ in the domain does not have an arrow leaving it.  It fails to be "everywhere defined"

If we change it in a different way, suppose that the domain is $\{a,b,c\}$ and the codomain is $\{\text{apple, banana, blueberry, cherry}\}$ with the same relation: a letter is related to a word iff the letter begins that word.  We have:

This fails to be a function because there is more than one arrow leaving $b$.  It fails to be "well defined."

Notice that in the above example, the relation did not change at all.  The only things that changed were the domain and the codomain.  In the same way, the examples in your question will either be or not be functions depending upon what the domain and codomain are.  The domain and codomain are only what they tell you they are, an image of the graph by itself will not tell you what they intend the domain or codomain to be.  Even though the image only pictures non-negative numbers, the problem statement implies that the domain is in fact $\Bbb R$.
If the domain is all real numbers and the codomain is a subset of the reals (possibly all reals), then it fails to "have an arrow leaving every vertex", it won't be everywhere defined, because there are numbers such as $-10$ for which no real number squared will equal.  Alternatively worded, $\sqrt{-10}$ is non-real.  (same issue as when I added the letter $d$ to the domain)
If the domain is non-negative real numbers and the codomain is all real numbers, then for the relation $\{(x,y)~:~x=y^2\}$ it is not a function because "some vertices have more than one arrow leaving them", it is not well defined, because there are numbers like $4$ which have multiple outputs.  $4=(-2)^2$ and also $4=2^2$ so both $(4,2)$ and $(4,-2)$ are included in the relation.  (same issue as when I added the word blueberry to the codomain)
If the domain was non-negative real numbers and the codomain is non-negative real numbers, then it will indeed be a function in both problems.  If the domain was non-negative real numbers and the codomain is all real numbers, only the first will be a function since $\sqrt{x}$ only outputs non-negative numbers there will be only one arrow leaving each.


Converting Linq2SQL simple queries to CompiledQueries to increase app performance

Question: I am writing a Silverlight for Windows Phone (SDK 7.1) app and I am displaying data from a CompactSQL DB in a LongListSelectorcontrol from the Silverlight Toolkit for Windows Phone.
Once the list becomes about 150 items long, The app really slows down loading data, navigating to and from pages and animations fail to display (I know using a background thread would help with freeing up the UI thread for animations).
I currently have three queries that I use constantly - everytime the data from LongListSelector is updated or the page is NavigatedTo. I have converted MoviesByTitle into a CompiledQuery and that has helped quite a lot, so I was looking to do the same for my other two queries(groupedMovies and LongListSelector.ItemSource of type List<Group<Movie>>), however I cannot seem to figure out the correct syntax.
Any suggestions on how I might make these queries more efficient - Through the use of CompiledQuery or otherwise?
MoviesByTitle is in another Class called Queries
public static Func<MovieDBContext, IOrderedQueryable<Movies>> MoviesByTitle = CompiledQuery.Compile((MovieDBContext db) => from m in db.Movies orderby m.Title,m.Year select m);

Fields in MainPage
private static List<String> characters = new List<String> { "#", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z" };
public static List<Group<Movies>> emptyGroups = new List<Group<Movies>>();

inside a LoadDB() method in MainPage - this method is called in OnNavigatedTo and in several other places when the DB is updated.
//Populates the 'empty' Groups of Movies objects only once.
if (emptyGroups.Count < 1)
{
    characters.ForEach(x => emptyGroups.Add(new Group<Movies>(x, new List<Movies>())));
}

IEnumerable<Movies> query = Queries.MoviesByTitle(App.db);

//Groups the objects
IEnumerable<Group<Movies>> groupedMovies = (from t in query
                                            group t by t.GroupHeader into grp
                                            orderby grp.Key
                                            select new Group<Movies>(grp.Key.ToString(), grp));

//Joins the 'empty' group and groupedMovies together for the LongListSelector control
moviesLongList.ItemsSource = (from t in groupedMovies.AsEnumerable().Union(emptyGroups)
                              orderby t.Title
                              select t).ToList();

GroupHeader is a property of Movies and an entity in the DB
[Column(CanBeNull=true, UpdateCheck = UpdateCheck.Never)]
public char GroupHeader
    {
        get
        {
            char l;
            //For Alphabetized Grouping, titles do NOT start with "The ..."
            if (this.Title.ToLowerInvariant().StartsWith("the "))
            {
                l = this.Title.ToLowerInvariant()[4];
            }
            else
            {
                l = this.Title.ToLower()[0];
            }
            if (l >= 'a' && l <= 'z')
            {
                return l;
            }
            return '#';
        }
        set { }
    }

The Group class is as follows
public class Group<T> : IEnumerable<T>
{
    public Group(string name, IEnumerable<T> items)
    {
        this.Title = name;
        this.Items = new List<T>(items);
    }

    public string Title
    {
        get;
        set;
    }

    public IList<T> Items
    {
        get;
        set;
    }
    ...
}

Answer: I assume that GroupHeader is an entity stored in the DB with 1-n relationship to the Movie entity.
First of all, I don't see 3 DB queries here. A LINQ expression is not always a DB query (e.g. there's LINQ to Objects). Sometimes determining what really is going on is quite challenging. The best friend in such cases is a DB profiler tool or the IntelliTrace - they show what queries are being executed on the DB at run time.
As far as I understand the code, you actually have 1+N queries: the first is MoviesByTitle, and then you have N queries in the expression that gets the movies grouped by their headers. It's N instead of 1 because you cast query to IEnumerable<> which makes it no longer a query but a simple CLR object, which is simply iterated in a foreach-like loop sending queries to the DB each time it needs a GroupHeader entity (it's an entity, isn't it?).
Try to combine 2 queries into one. You might even not need to use CompiledQuery. Here's an approximate code:
// I left this without changes
if (emptyGroups.Count < 1)
{
    characters.ForEach(x => emptyGroups.Add(new Group<Movies>(x, new List<Movies>())));
}

// I put var here, but it actually is an IQueryable<Movie>
var query = from m in App.db.Movies orderby m.Title, m.Year select m;

// Here var is IQueryable<anonymous type>, I just can't use anything else but var here
var groupedMoviesQuery = from t in query
                    group t by t.GroupHeader into grp
                    orderby grp.Key
                    select new
                    {
                       Movies = grp,
                       Header = grp.Key
                    };

// I use AsEnumerable to mark that what goes after AsEnumerable is to be executed on the client
IEnumerable<Group<Movie>> groupedMovies = groupedMoviesQuery.AsEnumerable()
                                            .Select(x => new Group<Movie>(x.Header, x.Movies))
                                            .ToList();

//No changes here
moviesLongList.ItemsSource = (from t in groupedMovies.AsEnumerable().Union(emptyGroups)
                              orderby t.Title
                              select t).ToList();

This code should work way better. What I actually did is that I turned your query from IEnumerable which is an iterate-only CLR object to an IQueryable which can be further wrapped into a more complex query. Now there's only one query which gets all movies grouped by headers. It should be fast.
I would introduce even more improvements to the code to make it work even faster:

You use Union of entities read from the DB and of some default list. You order it afterwards. You can safely remove all other orderings from your Linq2Sql code ('query' and 'groupedMoviesQuery')
Grouping seems to be not the most efficient here as compared to a join. Why don't you just query GroupHeaders including their related Movies? That should produce a JOIN in the db query which should be more efficient than a GROUP BY.
If you still have issues with performance, you can turn the query into a compiled query.

I'll show an example of a compiled query for you original logic with optimization of the first item of the list above:
class Result
{
  public GroupHeader Header {get;set;}
  public IEnumerable<Movie> Movies {get;set;}
}

public static Func<MovieDBContext, IQueryable<Result>> GetGroupHeadersWithMovies =
  CompiledQuery.Compile((MovieDBContext x) =>
      from m in x.Movies
      group m by m.GroupHeader into grp
      select new Result
      {
        Movies = grp,
        Header = grp.Key
      });

Comment: Thanks for all that. Especially for the explanation - the knowledge is going to help me more in the long run than straight out code.

I used Stopwatch to test my code and then implemented some of your suggestions where optimization was needed.


A question about the proof of Gray's theorem in contact geometry in da Silva's notes

Question: I'm reading Lecture 10 of Lectures on Symplectic Geometry by Ana Cannas da Silva. Here is the online version.
The theorem in question is Theorem 10.5 on page 59. I'll use the notation from the proof. I'm asking because I'm unable to check that the constructed $\rho_t$ indeed gives the answer. To show the existence of $u_t$ such that $\rho_t^*\alpha_t=u_t\cdot\alpha_0$, it is necessary and sufficient to show that $\rho_t^*\alpha_t$ vanishes on $H_0=\ker\alpha_0$. Thus for $v\in H_0$ we consider $\frac{d}{dt}(\rho^*_t\alpha_t)=\rho_t^*(\mathcal{L}_{v_t}\alpha_t+\frac{d\alpha_t}{dt})$, and we wish to show that it is $0$. However, by contruction we only know that $\rho_t^*(\mathcal{L}_{v_t}\alpha_t+\frac{d\alpha_t}{dt})$ vanishes on $\rho_t^*H_t$, hence we would need $\rho_t^*H_t=H_0$ to finish the proof. But this is just what we set out to prove in the beginning, so the argument doesn't work.
What is the correct missing detail in the proof? Thank in advance!
Answer: I'm not sure if the reasoning is correct. You don't need to prove that $\frac{d}{dt}(p_t^*\alpha_t)=0$. You aim to prove that $p_t^*\alpha_t=u_t \alpha_0$, which implies that $\frac{d}{dt}(p_t^*\alpha_t)=d(u_t \alpha_0)$. When trying to solve the differential equation, if you impose that your hypothetic solution $X_t$ is lying in $H_t$ the equation reduces to
$$ \iota_{X_t}d\alpha_t|_{H_t}=-\frac{d\alpha_t}{dt}|_{H_t}.  $$
Sinde $d\alpha_t$ is nondegenerate in $H_t$, the vector field $X_t$ is uniquely determined and you have a solution. By integrating it you obtain $p_t$ and deduce the expression of $u_t$ from the equation $p_t^*
\alpha_t= u_t \alpha_0$. By construction you have $p_t^*\alpha_t=u_t\alpha_0$ which implies that $p_t^*H_t=H_0$.
Comment: How can you "deduce the expression of $u_t$ from the equation $p_t^\ast\alpha_t=u_t\alpha_0$"? The existence of such $u_t$ is equivalent to what we what to show.
Comment: @Simplyorange We know that the equation $$ \iota_{X_t}d\alpha_t|_{H_t}=-\frac{d\alpha_t}{dt}|_{H_t}.  $$ has a solution by the non-degeneracy of $d\alpha_t$ along $H_t$. So we know $X_t$ exists (and is unique in fact). By integrating this time dependent vector field, we get a flow $p_t$. This flow satisfies $p_t^*\alpha_t=f_t.\alpha_0$ for some function $t$-dependent function $f_t$, by construction. What I meant by "deduce" is that this $f_t$ is exactly $u_t$: you simply factor out $\alpha_0$ from the expression of $p_t^*\alpha_t$.


Getting $head_title in my module

Question: I am building a module that requires me to get the $head_title variable in the hook_node_view for some processing...I have found info about that variable in relation to themes and around core, but not getting it in a module...I have successfully used drupal_get_title(), and variable_get() to get info, but I can not figure out how to get $head_title...
Any help?
Answer: $head_title is one of the variables passed to the html.tpl.php template file, and set from template_preprocess_html().
  // Construct page title.
  if (drupal_get_title()) {
    $head_title = array(
      'title' => strip_tags(drupal_get_title()),
      'name' => check_plain(variable_get('site_name', 'Drupal')),
    );
  }
  else {
    $head_title = array('name' => check_plain(variable_get('site_name', 'Drupal')));
    if (variable_get('site_slogan', '')) {
      $head_title['slogan'] = filter_xss_admin(variable_get('site_slogan', ''));
    }
  }
  $variables['head_title_array'] = $head_title;
  $variables['head_title'] = implode(' | ', $head_title);

That variable would be available as $variables['head_title'], if your module implements hook_preprocess_html() or hook_process_html(). Otherwise, you just need to execute similar code to the one used by template_preprocess_html(). To adapt that code for a module, outside hook_preprocess_html() and hook_process_html(), I would rewrite it as follows.
  // Construct page title.
  if (drupal_get_title()) {
    $head_title = array(
      'title' => strip_tags(drupal_get_title()),
      'name' => check_plain(variable_get('site_name', 'Drupal')),
    );
  }
  else {
    $head_title = array('name' => check_plain(variable_get('site_name', 'Drupal')));
    if (variable_get('site_slogan', '')) {
      $head_title['slogan'] = filter_xss_admin(variable_get('site_slogan', ''));
    }
  }

  // Remove the following line if you just need the string.
  $head_title_array = $head_title;

  $head_title = implode(' | ', $head_title);

Getting $head_title in my module


onsenui: Slidemenu width is not working right when screen width change (portrait to landscape)

Question: When I view app in portrait mode and slide left to right, the above page cover around 10% on the right edge of below page. And i rotate screen to landscape mode, the above page cover 50% of the below page. How can i make it consistent between portrait and landscape mode?
Answer: You can use max-slide-distance.
<code><ons-sliding-menu max-slide-distance="50%">
</code>
Comment: Thanks. But it will cover the contain in below page on small screen device, if the contain in below page width is larger than 50% of the page.


How to perform a repeated measure CFA

Question: My purpose is to verify that a specific scale is unidimensional, and that the one dimensional structure is persistent over time. I have a 10 item scale, about 200 cases and score values at two different time points.
It would be simple if it was only one measure: I would split the data in half, I would perform PCA in one half and CFA in the other half, an approach that indeed is working well for the first measure. However, I would like to somehow provide a measure that the unique factor structure is persistent over time.
How can this be done?
Answer: You need to do a CFA with two factors - a factor for time 1 and a factor for time 2. Each factor is keyed to its respective items.
You need to fit two models. In Model 1, you do a basic CFA where each regession coefficient can seek its natural level. Factors 1 and 2 are correlated. If Model 1 fails, you might want to try Model 1a, by adding correlations between the error terms of corresponding items.
Now fit Model 2, to test for persistence over time.
In Model 2, constrain the regressions for each item to be the same at both time periods. Test Model 2 against Model 1. If Model 2 fits as well as Model 1 (Difference of deviances not significant under chi-squared), you can assume that you have a unidimensional factor at both time periods with the same relationship to the items on the measure.
Here it is using the usual model formulations:
For definiteness, consider item 2. At time 1.$$ F_1=\lambda_{21}x_{21}+ \epsilon_{21}$$ For item 2 at time 2. $$ F_2=\lambda_{22}x_{22}+ \epsilon_{22}$$. For Model 1a, posit $Cov(\epsilon_{21}, \epsilon_{22}) \neq 0$. For Model 1, these error terms are indepdendent.
Using this notation, in Model 2, you posit $\lambda_{21}=\lambda_{22}$, and likewise for all regression coefficients.
Comment: I am obliged for your proficient answer. It seems complicated at first but I guess that I will manage to implement it in R.
Comment: Are you using lavaan?
Comment: Actually, I was thinking for sem package. Are you suggesting lavaan instead? Further, is there any bibliographic reference or other recommendation for this procedure to cite in an article?
Comment: Both packages are good. Yves Rousseel, of the lavaan package, has a good article in the journal of statistical software about the method and how to use lavaan. You can find a number of resources and references at the lavaan site, http://lavaan.ugent.be/
Comment: Indeed, it seems that lavaan is better supported than sem and I will use it in my problem. Thanks again for your help.

How to perform a repeated measure CFA