merge sort merge run time anaylsis

Question: I have some confusion of the run time analysis of the merge function in merge sort.
Merg(A,p,q,r)
1 n1=q-p+1
2 n2=r-q
3 let L[1..n1+1] and R[1..n2+1] be new arrays
4 for i=1 to n1
5     L[i] = A[p+i-1]
6 for j =1 to n2
7    R[j] = A[q+j]
8  L[n1+1] = infinity
9 R[n2+1] = infinity
10 i =0
11 j=0
12 for k=p to r
13   if L[i]<=R[j]
14       A[k]=L[i]
15       i=i+1
16    else A[k] = R[j]
17       j=j+1

In my book it says the following: to see that the merge procedure runs in O(n) time, where n=r-p+1, observe that each of lines 1-3 and 8-11 takes constant time, the for loops of lines 4-7 take O(n1+n2) = O(n)time, and there are n iterations of the for oop of lines 12-17, each of which takes constant time
My question is why do lines 12-17 take constant time per iteration and not affect the run time but
lines 4-7 don't take constant time. to me, it seems like both loops are doing the same thing. can someone help clarify this for me? Thanks!
Answer: Lines 1 through 9 are just splitting the input (A) into two pieces (L and R). Lines 10 and 11 are doing a bit of initializing to get ready for merging. The merge itself is from lines 12-17.
IOW, everything before line 12 (or arguably 10, not that it really matters) is irrelevant to analyzing the merge because none of it is really part of the merge at all.
Edit: Ultimately, however, a single iteration from 1 to 27 is linear: in lines 4-7, you walk exactly once through the input array, assigning each input exactly once to either L or R. In lines 12-27, you then walk through those two pieces, and copy them back to the original input. Ignoring a few other minor details like initializing i and j, the total number of operations is exactly 2N. For big-O notation, constant factors are ignored, so it's O(N).
Comment: well, thats what i would assume too, but the book says that the O(n) run time comes from lines 4-7, not lines 12-27
Answer: It's confusingly written. Both loops (4-7 and 12-17) have the same length (n) and the inside of both loops are constant time (no nested loops). So they're each O(n), for a total of O(n) for the whole routine.
Regarding Jerry's answer, lines 4-7 matter because they're still O(n). If you could magically remove lines 12-17 you'd still have an O(n) routine.
Comment: OH WOW. im dumb lol. for some reason i was thinking of them as being nested, which would make it O(n^2) but it would really be 2n which is just O(n). the books explanation is definitely not written in the best way, but i understand it now. Thanks!

merge sort merge run time anaylsis


Best way to store table like data

Question: What is the best way to store table like data like:
         | Column 0 | Column 1 | Column 2 | Column 3 | Column 4
Row 0    | 0.984546 | 0.654564 | 0.467676 | 0.673466 | 50.03333
Row 1    | 0.684546 | 0.457564 | 0.467776 | 0.674566 | 45.73335
Row 2    | 0.884546 | 0.424564 | 0.445676 | 0.664566 | 12.23333
Row 3    | 0.284546 | 0.054564 | 0.237676 | 0.124566 | 45.01333

Both max column and row is dynamic and the coordinates of the value is very important, as I have to do calculation based on those value. It's not a database because the data is different every time and I don't want it to be forever there. Also, I need it to be easily accessible. I try to use LinkedList with Double[] but it's just too difficult for me to locate each of the value. Speed is a factor too.
Any suggestion on the type of collection I should use or ways to handle it?
Comment: It's not clear what your use cases are of your collection but it's possible this might be one of the few cases where you want to go with a double[,] over a generic collection
Comment: You could use a `Dictionary>` or a class wrapping it into an indexer.
Answer: I think you should declare this variable:
List< List<double> > matrix = new List< List<double> >();

Later you can use matrix.Capacity to decide the number of rows and use that same property on each row to set the number of columns. This way you wont have memory wasted.
You can use matrix.TrimExcess() to resize the matrix if your next data will be shorter.
The idea behind using matrix.Capacity and matrix.TrimExcess() is to have control of how much space is reserved in the data structure. This way it wont grow by itself and you will gain performance.
Comment: @SunnySunday Then you shouldn't go with `DataTable` or `XML` approach. Those are a lot slower than this. If you want something faster than this, your only option is `double[,] matrix`
Comment: You can use a DataTable or save them in file using Xml
Comment: @moez I think `DataTable` is not needed in this scenario, and saving to a file isn't a requirement of the question. I see no problem with my answer.
Comment: I'm just afraid this will make things slow down
Comment: Do you mean 2D array?
Answer: You can use the .NET DataTable class without using a database at all... You can build one yourself very easily without any datasource.
This way, you still get all of the benefits (you can query it with LINQ, bind it to stuff, still have it be dynamic) but you don't need it to be backed by a database.
Here is a SO answer on how to accomplish that. https://stackoverflow.com/a/1042638/643951
Answer: Another option is to use XML structure
Comment: I think that's overkill, dynamic grow data structure should do it just fine.

Best way to store table like data


Prevent moment.js accepting an integer as a valid ISO8601 date

Question: I have some API's for creating dashboard widgets.  Those API's return basic name/value data pairs that are passed to Google Charts.  Moment.js checks whether the value is an ISO8601 date, and if so passes to Google Charts as a date instance.
However, the ISO_8601 isValid check is currently returning true if the date is a simple integer, e.g. 1234:
var myInt = 1234;
if (moment(myInt, moment.ISO_8601, true).isValid()) {
    console.log("Valid!");
}

I couldn't locate the necessary functionality to force a date format in the moment.js code, so this brutal hack works for now:
var myInt = 1234;
if (JSON.stringify(myInt).includes("T") && moment(myInt, moment.ISO_8601, true).isValid()) {
    console.log("Valid!");
}

Is there a correct way to use moment.js to configure the isValid() check?
The date format from my API is yyyy-mm-ddThh:mm:ss (without Z on the end).
Answer: According to THIS answer, when using strict parsing (last parameter set to true) you should also specify parse format, to avoid situations like you discribed. As many users notice, specyfying string format instead of using moment.ISO_8601 works as expected.

alert(isISODateValid(123)); //false
alert(isISODateValid("2011-10-10T14:48:00")); //true
alert(isISODateValid("2011-10-10T14:48:00Z")); //true

function isISODateValid(date) {

  return moment(date.toString().replaceAll("Z",""), "YYYY-MM-DDTHH:mm:ss", true).isValid();

}

EDITS: Updated snippet - if date contains "Z" suffix, remove it before parsing  validating source date format
Comment: Hmm interesting, thanks.  The challenge here is that if the API returned the date *with `Z` suffix* - a totally valid ISO8601 format, then we have to remember to change the parsing format.  I think your solution with multiple `isValid` checks might be the solution... `if (moment(1234, "YYYY-MM-DDTHH:mm:ss", true).isValid() || moment(1234, "YYYY-MM-DDTHH:mm:ssZ", true).isValid()) `
Comment: You can also strip all incoming dates from the "Z" letter to match the format since "Z" only means that there is zero offset in the timestamp and changes nothing in the matter of parsing.
Comment: According to Wikipedia: https://en.wikipedia.org/wiki/ISO_8601

An offset of zero, in addition to having the special representation "Z", can also be stated numerically as "+00:00", "+0000", or "+00".
Comment: Good answer.  Thank you.  I'll award the bounty when the time limit allows.
Comment: Please, check above post edits. A simple function, that allows to minimize code needed for validation, also is free from unnecessary repetition
Answer: Since you stated that: "The date format from my API is yyyy-mm-ddThh:mm:ss (without Z on the end)", the best way to parse it is explictly pass the format you are expecting to moment, using the right moment format tokens instead of using moment.ISO_8601.
So, in your case, simply use moment(myInt, "YYYY-MM-DDTHH:mm:ss", true), as shown in the snipppet:

function checkValid(input) {
  if (moment(input, "YYYY-MM-DDTHH:mm:ss", true).isValid()) {
    console.log(input + " is valid!");
  }
}

checkValid(1234);
checkValid("2021-04-27T20:40:15");
checkValid("2021-04-27T20:40:15Z");

Please note that the Z at the end stands for the timezone offset UTC+0, if you have it moment takes it into account, while without it, moment parses the input as local time (see Local vs UTC vs Offset guide)
As a side note, moment.ISO_8601 works as you were expecting in moment versions prior to 2.25.0:

function checkValid(input) {
  if (moment(input, moment.ISO_8601, true).isValid()) {
    console.log(input + " is valid!");
  }
}

checkValid(1234);
checkValid("2021-04-27T20:40:15");
checkValid("2021-04-27T20:40:15Z");

Answer: You could testify the sting before you pass it to the moment.? I have taken an example from this post
/**
 * RegExp to test a string for a full ISO 8601 Date
 * Does not do any sort of date validation, only checks if the string is according to the ISO 8601 spec.
 *  YYYY-MM-DDThh:mm:ss
 *  YYYY-MM-DDThh:mm:ssTZD
 *  YYYY-MM-DDThh:mm:ss.sTZD
 * @see: https://www.w3.org/TR/NOTE-datetime
 * @type {RegExp}
 */
var ISO_8601_FULL = /^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?(([+-]\d\d:\d\d)|Z)?$/i

// Usage:

ISO_8601_FULL.test( "2016-05-24T15:54:14.876Z" )  // true
ISO_8601_FULL.test( "2002-12-31T23:00:00+01:00" ) // true
ISO_8601_FULL.test( "2016-02-01" )                // false
ISO_8601_FULL.test( "2016" )                      // false

if (ISO_8601_FULL.test(myDate) && moment(myDate, moment.ISO_8601, true).isValid()) {
    console.log("Valid!");
}

I suppose the date should not be an integer.
Comment: What about the concrete value of the question `1234` (as number, not as string)?
Comment: This is much too complicated and very expensive in terms of performance.
Answer: Interesting is the difference between string and number. If it's a number, it is interpreted as the number of milliseconds since epoc, which is quite helpful in computer languages, but obviously not always what was requested and also not obvious to every developer. This can easily avoided with a type check (typeof input != 'string').
The other variant is more confusing: "12345" is not valid. good. But "1234" is interpreted as a year, and at the same time "34" seems to be interpreted as a time offset in minutes (Sun Jan 01 1234 00:00:00 GMT+0034). To me, this clearly looks like a bug in the library, since it's quite useless to parse the same digits multiple times for different purposes. But also after this is fixed, "1234" stays a valid date (year only) as defined in the standard ISO 8601
https://en.wikipedia.org/wiki/ISO_8601

For reduced precision,[17] any number of values may be dropped from any of the date and time representations, but in the order from the least to the most significant. For example, "2004-05" is a valid ISO 8601 date, which indicates May (the fifth month) 2004. This format will never represent the 5th day of an unspecified month in 2004, nor will it represent a time-span extending from 2004 into 2005.

Btw: "543210" is also valid and means "5432-10", or October of the year 5432

function checkValid(input) {
  m = moment(input, true);
  console.log(input + "  type: " + typeof input + "  valid: " + m.isValid() + "  result:" + m.toString());
}

checkValid(1234);
checkValid("1234");
checkValid(12345);
checkValid("12345");
checkValid("2021-04-27T20:40:15");
checkValid("2021-04-27T20:40:15Z");


Can't upgrade docker on rackspace

Question: I have a rackspace docker-machine set up and I need to upgrade it from 1.9 to 1.10. Using the docker-machine upgrade command says that it upgrades the machine, but docker-machine ls remains showing 1.9. If I spin up a new rackspace docker-machine it correctly shows at 1.10.
Does anyone know how to correctly upgrade a rackspace driven docker-machine?
Here is my current docker toolset version information:
$ docker -v
Docker version 1.10.3, build 20f81dd
$ docker-machine -v
docker-machine version 0.6.0, build e27fb87
$ docker-compose -v
docker-compose version 1.6.2, build 4d72027

Comment: not sure how docker-machine works other then on my laptop but did you try restarting docker and seeing what a `docker version` says?
Comment: Tried a full computer restart.  I even created a new rackspace instance that came up at 1.10
Comment: did you tried upgrade it via your package manager? are there any errors ? what distribution of OS do you run ?
Comment: I'm not sure what you mean by update it via the package manager.  I am attempting to manage this server from an OSX 10.11.4 machine, but the actual server is running whatever Rackspace uses
Answer: The commands you are running are showing you the version of the software installed on your machine, the latest version of docker-machine is docker-machine version 0.6.0, build e27fb87
To ensure that the docker service is up to date run the following docker-machine ssh then run one of the following (I believe the dpkg command) rpm -qa | grep docker OR dpkg -l | grep docker, this will tell you what the installed version of Docker is on the server.
Comment: `root@XXt:~# dpkg -l | grep docker
rc  docker-engine                       1.9.1-0~trusty                amd64        Docker: the open-source application container engine
ii  lxc-docker-1.9.1                    1.9.1                         amd64        Linux container runtime`
Comment: what happens if you try to do a `apt-get update && apt-get upgrade` ?
Comment: After both updates dpkg results haven't changed
Comment: OK, lets try `apt-get install docker` ?
Comment: Now I get this: `root@XX:~# dpkg -l | grep docker
ii  docker                              1.5-1                            amd64        System tray for KDE3/GNOME2 docklet applications
rc  docker-engine                       1.9.1-0~trusty                   amd64        Docker: the open-source application container engine
rc  lxc-docker-1.9.1                    1.9.1                            amd64        Linux container runtime`

Can't upgrade docker on rackspace


Get fields from a collection field in .theme

Question: I would like to know how I have to collect the field collection to get the fields it contains
$fieldItemList = $node->field_collection_gallery;
$fieldCollection = $fieldItemList[0];
$fieldCollectionItem = FieldCollectionItem::load($fieldCollection->value);

Comment: Use `->target_id` instead of `->value`. Now that you have the field collection item it should be no problem to access the fields. You can use `$node->field_collection->referencedEntities()`, this is quicker
Answer: Use the following code:
$fieldCollectionItems = $node->get('field_collection_gallery');

    foreach ($fieldCollectionItems as $key => $item) {
      $fcItem = FieldCollectionItem::load($item->value);

      if ($fcItem instanceof FieldCollectionItem) {
        $fcValue = $fcItem->get('field_name')->getValue();
      }
    }

The $fcValue variable should contain the value from the field in the field collection item.
You have to replace the field_name with your own field name in the last line in the foreach() loop.


Text Classification using Decision Trees in Python

Question: I am new to Python as well as machine learning. My implementation is based on the IEEE research paper http://ieeexplore.ieee.org/document/7320414/ (Bug report, feature request, or simply praise? On automatically classifying app reviews)
I want to classify text into categories. The text is user reviews from google play store or apple app store. The categories used in the research were Bug, Feature, User Experience, Rating. Given this situation, I am trying to implement a decision tree using sklearn package in python. I came across an example data set provided by sklearn 'IRIS', which builds a tree model using the features and their values mapped to the target. In this example, it is numeric data.
I am trying to classify text instead of numeric data. Examples:

I liked very much the upgrade to pdfs. However, they aren't displaying anymore Fix it and it will be perfect [BUG]
I just wish it would notify me if I go below a certain dollar amount [FEATURE]
This app is very helpful in my line of business [Rating]
Easy to find songs and purchase in iTunes [UserExperience]

Given these texts and lot more user reviews of these categories, I want to create a classifier that can train using the data and predict the target of any given user reviews.
So far I have pre-processed the text and created training data in the form of list of tuples that contains the pre-processed data and its target.
My Pre-processing:

Tokenize multi-line comments to into single sentences
Tokenize each sentence into words
Remove stop words in the tokenized sentence
Lemmatize the words in the tokenized sentence

(['i', 'liked', 'much', 'upgrade', 'pdfs', 'however', 'displaying', 'anymore', 'fix', 'perfect'], "BUG")
Here's what I have so far:
import json
from sklearn import tree
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import sent_tokenize, RegexpTokenizer

# define a tokenizer to tokenize sentences and also remove punctuation
tokenizer = RegexpTokenizer(r'\w+')

# this list stores all the training data along with it's label
tagged_tokenized_comments_corpus = []

# Method: to add data to training set
# Parameter: Tuple in the format (Data, Label)
def tag_tokenized_comments_corpus(*tuple_data):
tagged_tokenized_comments_corpus.append(tuple_data)

# step 1: Load all the stop words from the nltk package
stop_words = stopwords.words("english")
stop_words.remove('not')

# creating a temporary list to copy the existing stop words
temp_stop_words = stop_words

for word in temp_stop_words:
if "n't" in word:
    stop_words.remove(word)

# load the data set
files = ["Bug.txt", "Feature.txt", "Rating.txt", "UserExperience.txt"]

d = {"Bug": 0, "Feature": 1, "Rating": 2, "UserExperience": 3}

for file in files:
input_file = open(file, "r")
file_text = input_file.read()
json_content = json.loads(file_text)

# step 3: Tokenize multi sentence into single sentences from the user comments
comments_corpus = []
for i in range(len(json_content)):
    comments = json_content[i]['comment']
    if len(sent_tokenize(comments)) > 1:
        for comment in sent_tokenize(comments):
            comments_corpus.append(comment)
    else:
        comments_corpus.append(comments)

# step 4: Tokenize each sentence, remove stop words and lemmatize the comments corpus
lemmatizer = WordNetLemmatizer()
tokenized_comments_corpus = []
for i in range(len(comments_corpus)):
    words = tokenizer.tokenize(comments_corpus[i])
    tokenized_sentence = []
    for w in words:
        if w not in stop_words:
            tokenized_sentence.append(lemmatizer.lemmatize(w.lower()))
    if tokenized_sentence:
        tokenized_comments_corpus.append(tokenized_sentence)
        tag_tokenized_comments_corpus(tokenized_sentence, d[input_file.name.split(".")[0]])

# step 5: Create a dictionary of words from the tokenized comments corpus
unique_words = []
for sentence in tagged_tokenized_comments_corpus:
for word in sentence[0]:
    unique_words.append(word)
unique_words = set(unique_words)

dictionary = {}
i = 0
for dict_word in unique_words:

dictionary.update({i, dict_word})
i = i + 1

train_target = []
train_data = []
for sentence in tagged_tokenized_comments_corpus:
train_target.append(sentence[0])
train_data.append(sentence[1])

clf = tree.DecisionTreeClassifier()
clf.fit(train_data, train_target)

test_data = "Beautiful Keep it up.. this far is the most usable app editor..
it makes my photos more beautiful and alive.."

test_words = tokenizer.tokenize(test_data)
test_tokenized_sentence = []
for test_word in test_words:
    if test_word not in stop_words:
     test_tokenized_sentence.append(lemmatizer.lemmatize(test_word.lower()))

#predict using the classifier
print("predicting the labels: ")
print(clf.predict(test_tokenized_sentence))

However, This doesn't seem to work since it throws an error during run time when we train the algorithm. I was thinking If I can map the words in the tuple to the dictionary and convert the text into numeric form and train the algorithm. But I am not sure if this may work.
Can anyone suggest how can I fix this code? Or if there is any better way to implement this decision tree.
Traceback (most recent call last):
  File "C:/Users/venka/Documents/GitHub/RE-18/Test.py", line 87, in <module>
clf.fit(train_data, train_target)
  File "C:\Users\venka\Anaconda3\lib\site-packages\sklearn\tree\tree.py", line 790, in fit
X_idx_sorted=X_idx_sorted)
 File "C:\Users\venka\Anaconda3\lib\site-packages\sklearn\tree\tree.py", line 116, in fit
X = check_array(X, dtype=DTYPE, accept_sparse="csc")
File "C:\Users\venka\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 441, in check_array
"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[ 0.  0.  0. ...,  3.  3.  3.].
Reshape your data either using array.reshape(-1, 1) if your data has a
single feature or array.reshape(1, -1) if it contains a single sample.

Comment: Do you have the traceback of the error that occurs? We can't exactly help you without knowing what actually went wrong. That said, I'm going to assume you might struggle plugging sentences straight into sklearn Decision Trees.
Comment: I have updated the post with the Traceback. I understand that it is very difficult to plug sentences straight into decision trees. I was thinking to create a dictionary taking all the unique words from the corpus and assign a unique numeric value to each word so that we can pass this as tuple of numeric data instead of sentence but not sure if this might work
Comment: Have you tried it? Why do you think it won't work? The error here is reasonably self explanatory, you need to hand in a 2d array. Is the one errorring your data or target?
Comment: I am yet to try it. In the IRIS data set example that I've mentioned in the post, the training data consists of constant 4 features for every singe data and I am assuming that these 4 values make it easy for the algorithm to understand and construct a model. However, when I convert text into numerical data, each sentence is not going to be of same length. I agree that this question exists even now without converting into numerical data. And regarding the error, I am looking into the stack trace again. It makes more sense now after you've mentioned and I am gonna try again.
Comment: You can use CountVectorizer or TfidfVectorizer in sklearn to convert your filtered words into numerical data ready to be used by ML algorithms.
Comment: Thank you for the suggestion, I did the same now to convert my textual data to numerical data
Answer: Decision trees can only work when your feature vectors are all the same length. Personally I've got no clue as to how effective Decision Trees would be at text analysis like this, but if you're to try and go for it, the way I'd suggest is a "one-hot" "bag of words" style vector.
Essentially, keep tag of how many times words appear in your example, and put them in a vector that represents the whole corpus. Say, once you removed all the stop-words the set of the entire corpus was:
{"Apple", "Banana", "Cherry", "Date", "Eggplant"}

You represent this by a vector the same size as the corpus, with each value representing whether or not the word appears. In our example, a 5 length vector where the first element is associated with "Apple", the second with "Banana" and so on. You might get something like:
bag("Apple Banana Date")
#: [1, 1, 0, 1, 0]
bag("Cherry")
#: [0, 0, 1, 0, 0]
bag("Date Eggplant Banana Banana")
#: [0, 1, 0, 1, 1]
# For this case, I have no clue if Banana having the value 2 would improve results.
# It might. It might not. Something you'd need to test.

This way, you have the same sized vector regardless of the input, and the decision tree knows where to look for certain outputs. Say "Banana" corresponds strongly to bug reports, in which case the decision tree will know that a 1 in the second element means a bug report is more likely.
Of course, your corpus might be thousands of words long. In that case, your decision tree probably won't be the best tool for the job. Not unless you first take some time to trim down your features.
Comment: Thanks for the answer. As expected and like you mentioned, Decision trees can work only when the feature vectors are all the same length. I came to understand that what you've suggested is to convert the text to a vector. This tutorial had a very good explanation http://blog.christianperone.com/2011/09/machine-learning-text-feature-extraction-tf-idf-part-i/ of behind the scenes of the sklearn's CountVectorizer and TfidfTransformer. The decision tree is done, I used the model_selection to split the data into train and test. I get an accuracy of 50.15%, however the Naive Bayes is giving me 58.30
Comment: I believe the accuracy can vary on the training data. For this accuracy that I've mentioned above, I had **Bug: 814, Feature: 744,  Rating: 3961, UserExperience: 1225** and split the train and test data as 60% and 40% respectively. This accuracy I achieved does not match the paper http://ieeexplore.ieee.org/document/7320414/ and their's is much more though. I will however continue to collect more data or choose a different way to pick the training data to improve the accuracy.


Ideal Settings for Longitudinal Models?

Question: The way I see it, logically speaking - Longitudinal Data (e.g. medical patients being measured repeatedly over a period of time) can have one of two forms:

Case 1: All patients are measured exactly "n" times
Case 2: Not all patients are measured exactly "n" times

(Also) Logically speaking, for a similar quality and volume of data -  I would imagine that applying Longitudinal Models in Case 1 must be "easier" than applying Longitudinal Models in Case 2. In this context, I imagine that "easier" refers to the (Longitudinal) model estimates "inherently" having better statistical properties such as lower asymptotic variance, higher efficiency and stronger consistency (I could be wrong about this).
As a crude and somewhat unrelated example, it is easier to stack books on top of each other if all books are roughly the same shape and size. If the books begin to vary in shape and size from each other, then this becomes more complicated. And if the books significantly vary in shape sand size from each other, then it becomes even more complicated to stack these books on top of each other. For instance, stacking 10 of the exact same textbooks on top of each other will likely be easier and likely result in a sturdier pile. Stacking 5 books of one size and 5 books of another size on top of each other will be more challenging and be less likely to be as sturdy (e.g. in the previous example, the 10 books of the same size can be stacked in any order and be as sturdy - whereas in this example, only some orderings of the books will somewhat sturdy and most of the orderings will not be as sturdy. For example, stacking the 5 bigger books on the bottom and the 5 smaller books on top will likely result in the sturdiest pile - but randomly stacking these 10 different sized books will less likely to be as sturdy). And if the books are significantly varying in sizes (e.g. stacking encyclopedias, dictionaries, magazines... audio cassette tapes), it will likely be very difficult to stack these books together and the resulting stacks will likely not be very sturdy. All in all, the more similar the books are to one another, the more easier they are to stack - the less similar the books are to one another, the less easier they are to stack.
My Question: Can this same analogy be extended to analyzing Longitudinal Data where all patients are measured the exact same number of times vs. Longitudinal Data where not all patients are measured the exact same number of times? Mathematically, can we somehow show that Longitudinal Models fitted to Longitudinal Data where all patients are measured the exact same number of times will inherently produce "higher quality estimates" compared to  Longitudinal Data where all patients are not measured the exact same number of times - or is this just a incorrect fallacy with no theoretical basis?
Thanks!
Additional Material: R Simulation
Using the R programming language, I loaded an (artificially created) Longitudinal Dataset :
# each row represents an a unique medical patient and each column represents the same variable being measured at 10 different points in time

 library(longitudinalData)
 data(artificialLongData)

I then randomly removed observations throughout this dataset (replaced them with NA) to represent some patients being measured less frequently than other patients:
    # First Dataset : All Patients With Equal Measurements
    first_dataset = artificialLongData

      id    t0    t1    t2    t3    t4    t5    t6    t7    t8    t9   t10
    1 s1  3.82  2.42 -1.85 -2.05  1.01  1.56  0.34  0.52 -0.06 -1.09  0.44
    2 s2 -4.88 -2.95 -2.38  3.73 -2.77  1.72 -0.99 -0.70  0.15  2.38 -0.72
    3 s3  0.57 -0.86  1.46 -2.04 -1.18  4.89 -3.94  0.50  4.90 -0.52  1.52
    4 s4 -3.83 -1.22  0.17 -0.22 -3.05 -0.74  1.93  1.85 -3.93 -2.91 -1.13
    5 s5 -3.99 -2.82 -2.13 -4.29 -2.33 -0.79 -0.86 -6.23 -8.13 -8.53 -0.87
    6 s6  6.67  5.87  8.45  4.80  3.78  4.18  4.16  2.10  3.14  3.46  5.49

    # Second Dataset : Some Patients (10%) are Measured Less Frequently

 id = first_dataset[,1]
 t10 = first_dataset[,12]

second_dataset = first_dataset[,-c(1,12)]

second_dataset = as.data.frame(lapply(second_dataset, \(x) replace(x, sample(length(x), .1*length(x)), NA)))

second_dataset = cbind(id, second_dataset, t10)

 id    t0    t1    t2    t3    t4    t5    t6    t7   t8    t9   t10
1 s1  3.82  2.42 -1.85 -2.05    NA    NA    NA    NA   NA    NA  0.44
2 s2    NA    NA -2.38  3.73    NA    NA -0.99 -0.70   NA    NA -0.72
3 s3    NA -0.86    NA    NA    NA  4.89    NA    NA   NA -0.52  1.52
4 s4 -3.83    NA  0.17 -0.22    NA -0.74  1.93    NA   NA -2.91 -1.13
5 s5    NA -2.82 -2.13 -4.29 -2.33 -0.79    NA -6.23   NA    NA -0.87
6 s6  6.67    NA    NA    NA  3.78    NA  4.16    NA 3.14    NA  5.49

# Third Dataset : Many Patients (50%) are Measured Less Frequently

id = first_dataset[,1]
t10 = first_dataset[,12]

third_dataset = first_dataset[,-c(1,12)]

third_dataset = as.data.frame(lapply(third_dataset, \(x) replace(x, sample(length(x), .5*length(x)), NA)))

third_dataset = cbind(id, third_dataset, t10)

  id    t0    t1    t2    t3    t4    t5    t6    t7    t8    t9   t10
1 s1    NA  2.42 -1.85    NA  1.01  1.56  0.34    NA -0.06 -1.09  0.44
2 s2    NA -2.95    NA    NA -2.77  1.72    NA -0.70    NA  2.38 -0.72
3 s3  0.57    NA    NA -2.04 -1.18  4.89 -3.94    NA  4.90    NA  1.52
4 s4 -3.83    NA    NA    NA -3.05    NA  1.93  1.85 -3.93 -2.91 -1.13
5 s5    NA    NA -2.13 -4.29 -2.33 -0.79    NA    NA -8.13 -8.53 -0.87
6 s6  6.67    NA    NA  4.80    NA  4.18    NA  2.10    NA  3.46  5.49

#visualization source: https://github.com/jevgenij-p/blog/blob/master/Missing%20Values%20Plot/MissingValues.R and https://jev-pankov.com/2017/11/15/visualize-missing-values-in-r/a

Suppose we are interested in fitting a Longitudinal Regression Model to these 3 datasets for predicting the value of "t10" for each patient. Although in my mind this is very a logical and natural result (it will be easier to "adapt" the longitudinal regression model to "fuller datasets" compared to the "emptier datasets) - can we somehow "mathematically prove" that in general, the properties of the Longitudinal Regression Models on data with conditions resembling the first dataset will be necessarily be "better" (e.g. lower asymptotic variance) than models on data with conditions resembling the second dataset, and both of these models will necessarily be "better" than models on data with conditions resembling the third dataset?
Thanks!
Answer: There is a simple counterexample to your suggestion. Imagine a case where some units are affected by large measurement error, while others have only small measurement error. It is more statistically efficient to collect more data from the units with greater measurement error than to collect the same amount from each unit.
Although the details depend on the model in question, here's a proof for a simple model. Say you have two patients, each patient's data are normally distributed with a patient-specific mean ($\mu_1$ and $\mu_2$) and standard deviation ($\sigma_1$ and $\sigma_2$), and you're interested in estimating the means. The error variance for each patient in this model is $\frac{\sigma_i^2}{n_i}$, where $n_i$ is the number of observations for patient $i$. You can collect $n = 100$ observations, and you want to minimise the total error variance, $\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{100 - n_1}$,
since this should also minimise the expected sum of squared errors.
Let's say $\sigma_1 = 1$ and $\sigma_2 = 4$.
We could find the value of $n_1$ that minimises the total error variance using calculus, but let's cheat and just use a computer.
total_squared_error = function(n1, n = 100, s1 = 1, s2 = 4){
  (s1^2 / n1) + (s2^2 / (n - n1))
}
plot(total_squared_error,
     from = 1, to = 50, xlab = 'Observations for patient 1', ylab = 'Total RSE')

This shows that the lowest expected total squared error
is achieved when $n_1 = 20$.
optimize(total_squared_error, c(1, 99))

$minimum
[1] 19.99999

$objective
[1] 0.25

Comment: @ Eoin: thank you for your answer!  Could you please write a bit more if you have time? Thank you!
Comment: What else would you like to know? The key point is that it's generally preferable to collect more data from the units that are harder to estimate parameters for, but the underlying maths differ depending on your design and analysis.

Ideal Settings for Longitudinal Models?


Proper Holomorphic Mappings

Question: Let $A,B$ denote domains in $\mathbb{C}$, and $f:A\rightarrow B$
is a holomorphic mapping. Suppose that $f$ is proper ($f^{-1}$ takes
compact sets to compact sets). Prove that $f(A)=B.$
Comment: $f$ must be non-constant...
Answer: A proper map is easily seen to be a closed map. Moreover, a non-constant holomorphic map on a domain in $\mathbb C$ is always an open map (see Wikipedia, Open Mapping Theorem for example).
Therefore, $f(A)$ is both open and closed (and non-empty of course), so by connectedness of $B$, $f(A)=B$.
Edit: Closedness of $f$.  If $f(x_n)\to y$, then $\{f(x_n), n \in \mathbb N\}\cup \{y\}$ is compact thus $\{x_n, n \in \mathbb N\}\subset f^{-1}(\{f(x_n), n \in \mathbb N\}\cup \{y\})$ lives in a compact set, therefore a subsequence of $(x_n)$ is convergent to some $x\in A$, satisfying of course $f(x)=y$, so $f$ is closed.

Proper Holomorphic Mappings


calculation threshold for minimum risk classifier?

Question: Suppose Two Class $C_1$ and $C_2$ has an attribute $x$ and has distribution $ \cal{N} (0, 0.5)$ and $ \cal{N} (1, 0.5)$. if we have equal prior $P(C_1)=P(C_2)=0.5$ for following cost matrix:
$L= \begin{bmatrix} 0  & 0.5 \\ 1 & 0 \end{bmatrix}$
why, $x_0 < 0.5$ is the threshold for minimum risk (cost) classifier?
This is my note example that I misunderstand, (i.e, how this threshold is reached? )
Edit 1: I think for thresholds of likelihood ratio we can use P(C1) / P(C2).
Edit 2: I add from Duda Book on Pattern some text about threshold.
Answer: For a cost matrix
$$L= \begin{bmatrix} 0  & 0.5 \\ 1 & 0 \end{bmatrix} \begin{matrix} c_1 \\ c_2 \end{matrix} \;\text{prediction} \\ \hspace{-1.9cm} \begin{matrix} c_1 & c_2 \end{matrix} \\  \hspace{-1.9cm}\text{truth}$$
the loss of predicting class $c_1$ when the truth is class $c_2$ is $L_{12} = 0.5$, and the cost of predicting class $c_2$ when the truth is class $c_1$ is $L_{21} = 1$. There is no cost for correct predictions, $L_{11} = L_{22} = 0$. The conditional risk $R$ for predicting either class $k$ is then
$$
\begin{align}
R(c_1|x) &= L_{11} \Pr (c_1|x) + L_{12} \Pr (c_2|x) = L_{12} \Pr (c_2|x) \\
R(c_2|x) &= L_{22} \Pr (c_2|x) + L_{21} \Pr (c_1|x) = L_{21} \Pr (c_1|x)
\end{align}
$$
For a reference see these notes on page 15.
In order to minimize the risk/loss you predict $c_1$ if the cost from the mistake of doing so (that's the loss of the wrong prediction times the posterior probability that the prediction is wrong $L_{12} \Pr (c_2|x)$) is smaller than the cost of wrongfully predicting the alternative,
$$
\begin{align}
L_{12} \Pr (c_2|x) &< L_{21} \Pr (c_1|x) \\
L_{12} \Pr (x|c_2) \Pr (c_2) &< L_{21} \Pr (x|c_1) \Pr (c_1) \\
\frac{L_{12} \Pr (c_2)}{L_{21} \Pr (c_1)} &< \frac{\Pr (x|c_1)}{ \Pr (x|c_2)}
\end{align}
$$
where the second line uses Bayes' rule $\Pr (c_2|x) \propto \Pr (x|c_2) \Pr (c_2)$. Given equal prior probabilities $\Pr (c_1) = \Pr (c_2) = 0.5$ you get
$$\frac{1}{2} < \frac{\Pr (x|c_1)}{ \Pr (x|c_2)}$$
so you choose to classify an observation as $c_1$ is the likelihood ratio exceeds this threshold. Now it is not clear to me whether you wanted to know the "best threshold" in terms of the likelihood ratios or in terms of the attribute $x$. The answer changes according to the cost function. Using the Gaussian in the inequality with $\sigma_1 = \sigma_2 = \sigma$ and $\mu_1 = 0$, $\mu_2 = 1$,
$$
\begin{align}
\frac{1}{2} &< \frac{\frac{1}{\sqrt{2\pi}\sigma}\exp \left[ -\frac{1}{2\sigma^2}(x-\mu_1)^2 \right]}{\frac{1}{\sqrt{2\pi}\sigma}\exp \left[ -\frac{1}{2\sigma^2}(x-\mu_2)^2 \right]} \\
\log \left(\frac{1}{2}\right) &< \log \left(\frac{1}{\sqrt{2\pi}\sigma}\right) -\frac{1}{2\sigma^2}(x-0)^2 - \left[ \log \left(\frac{1}{\sqrt{2\pi}\sigma}\right) -\frac{1}{2\sigma^2}(x-1)^2 \right] \\
\log \left(\frac{1}{2}\right) &< -\frac{x^2}{2\sigma^2} + \frac{x^2}{2\sigma^2} - \frac{2x}{2\sigma^2} + \frac{1}{2\sigma^2} \\
\frac{x}{\sigma^2} &< \frac{1}{2\sigma^2} - \log \left(\frac{1}{2}\right) \\
x &< \frac{1}{2} - \log \left(\frac{1}{2}\right) \sigma^2
\end{align}
$$
so a prediction threshold in terms of $x$ as you search for can only be achieved if the losses from false predictions are the same, i.e. $L_{12} = L_{21}$ because only then can you have $\log \left( \frac{L_{12}}{L_{21}} \right) = \log (1) = 0$ and you get the $x_0 < \frac{1}{2}$.
Comment: @whuber thanks, I completely missed that so I started from a completely wrong end.
Comment: Nice Answer, but confused me! if you want to choose $x_0=0.5$ or $x_0<0.5$, which one is correct?
Comment: So right on the decision boundary $x_0=0.5$ you can't tell exactly if an observation should be in class one or two (because it's exactly on the boundary). So choosing whether observation $i$ should be in class 1 if $x_0 \leq 0.5$ or $x_0 < 0.5$ is up to you. With large enough samples this should happen for very few observations so at the margin it will matter litter for your result.
Comment: all of my problem that set bounty to it thtat my prof. calculated $x_0<0.5$ and not accept $x_0=0.5$ please see my edit in question, I thin threshold should be $x_0<0.5$ .
Comment: maybe 0.5-ln :)
Comment: That was the point though of the picture. You see that below the black line (decision boundary) you can tell exactly that an observation should be in class one. To the right of the black line the observation should be in class two. You can't tell for observations directly on the line, hence they are in class one for sure if $x_0 < 0.5$.
Comment: you means threshold is exactly $x_0<0.5 $? it's tricky :)
Comment: I have looked through this answer several times but could find no reference at all to the cost matrix. How does it affect the solution?
Comment: please update asap to set bounty
Comment: I'm not sure what I'm supposed to update? The threshold for the minimum risk classifier for the given risk matrix is already derived in the answer together with the conditions required to find a threshold of 0.5
Comment: best is false. I correct it, please update your answer to set bounty.


What value should the $bundle_name parameter for fileld_info_instances() get?

Question: In a module, I need to get the list of fields for a content type. I used the following code, but $instances does not contain any helpful value.
$instances = field_info_instances('node', 'indicatorsheet_page');

Which value should I use for $content_type? Isn't it the content type machine name?
Answer: What you're doing here seems correct to me, bear in mind that you can leave the bundle parameter empty and then you'll get an array of results keyed by bundle so you can evaluate if there's a problem with the machine name of the bundle you're using:
  $instances = field_info_instances('node');

You can also access admin/reports/fields and take a look to fields and content types available in your site.
Answer: The second argument required from field_info_instances() is the machine name for a content type. If the function doesn't return any information when 'node' and 'indicatorsheet_page' are passed as arguments, that could mean:

The content type name isn't correct
The content type name is correct, but that isn't a node content type
The content type name is correct, and that is a node content type, but it doesn't have any field that isn't marked as deleted

Since the field information is cached, it could also be the cache_field cache has been corrupted.
For debugging purposes, you can check what field_read_fields(array('entity_type' => 'node'), array('include_deleted' => TRUE, 'include_inactive')) and field_read_instances(array('entity_type' => 'node'), array('include_deleted' => TRUE, 'include_inactive')). Both the function calls return information for fields of every node content type, including information for those fields have been marked as deleted or inactive fields.


Ajuda com operadores de comparação Javascript

Question: Boa Tarde,
Estou fazendo um trabalho, do qual vou ter diversos formulários com um numero minimo e máximo e estou querendo criar uma função de verifica e evita a inserção de números fora da faixa pelo teclado, a minha função ficou desta forma

$(':input[type="number"]').on('keyup', function(event) {
  var id = this.id;
  var min = document.getElementById(id).getAttribute('min');
  var max = document.getElementById(id).getAttribute('max');

  verificaTypeNumber(id, min, max);
});

function verificaTypeNumber(idInput, minValue, maxValue) {
  var aux = document.getElementById(idInput).value;

  if (aux == null || aux < minValue) {
    document.getElementById(idInput).value = minValue;
  } else if (aux > maxValue) {
    document.getElementById(idInput).value = maxValue;
  }

}

<input type="number" id="Teste" min="10" max="999" value="10">

Ao digitar um valor abaixo da faixa (no exemplo menor que "10") a função funciona e entra dentro do if(), porem para um valor acima da faixa (acima de 999), ela não funciona e não entra dentro do else if()
Answer: O .value de um input e os atributos min e max que estás a ler são strings. E por causa disso ele está a comparar texto e não numeros.
Repara na minha resposta que uso Number para converter texto (strings)  em numeros e assim a tua lógica já funciona.
Mudei também o que é passado à função, assim em vêz de chamares document.getElementById(idInput) 5 vezes, não chamas nenhuma e usas o this que é o elemento que queres.

$(':input[type="number"]').on('keyup', function(event) {
  var min = Number(this.getAttribute('min'));
  var max = Number(this.getAttribute('max'));

  verificaTypeNumber(this, min, max);
});

function verificaTypeNumber(el, minValue, maxValue) {
  var aux = Number(el.value);

  if (aux == 0 || aux < minValue) {
    el.value = minValue;
  } else if (aux > maxValue) {
    el.value = maxValue;
  }

}

<input type="number" id="Teste" min="10" max="999" value="10">


GPS oscilando mesmo estando em local fixo

Question: Estou trabalhando em um app Ionic, com rastreabilidade, mas o sinal de gps oscila muito certa de 200m, mesmo estando em um lugar fixo, conhecem alguma solução para isso?
Comment: Não tem como você saber se de fato está parado só pelo GPS. Uma possibilidade seria fazer um "arredondamento" de leituras anteriores, mas aí teria um atraso no movimento. Poderia condicionar o arredondamento a um acelerômetro (se não tiver aceleração por determinado tempo, considere que está parado). Lembrando que nesse caso, o simples fato da pessoa manusear um telefone (se o GPS for um telefone, claro) seria detectado como movimento. Ajudaria você [edit] a pergunta e dar um pouco mais de detalhes do que sua aplicação vai fazer e o tipo de GPS, para sugestões mais específicas.
Comment: Em ambientes fechados é normal o GPS oscilar. Ele tem mais precisão em locais abertos.
Comment: @DanielOmine, sua afirmação não está correta. Smartphones possuem localização por GPS sim. A localização pelas antenas é mais uma forma de derivar a localização, mas a que tem mais precisão é a do GPS. Baixe uma app chamada GPS Test no Google Play, ela mostra a posicao e o sinal recebido dos satelites de GPS. Sobre o GPS oscilar mesmo estando parado, é completamente normal: http://www.gpstrackersecurity.com/why-gps-drift-when-stationary/. Como o Bacco disse, a aplicação precisa lidar com isso de alguma forma, mas de longe é um problema trivial de resolver.
Comment: @LuizVieira, eu escrevi errado, queria dizer que de longe *não* é um problema trivial de resolver. Desculpe-me.
Comment: Sabe alguma forma de eu simular um evento "estacionado" e voltar a capturar quando estiver em movimento novamente?
Comment: Pode ser que tenha uma forma mas eu particularmente desconheço, nunca tentei fazer algo assim.
Comment: Normalmente um smartphone não possui GPS. A localização se dá pelas antenas de telefonia móvel. Cada antena possui GPS e por meio de cálculos, que popularmente conhecemos por "triangulação de antenas", é determinado o local aproximado do dispositivo. E note que nem sempre é um local preciso, mesmo estando parado. Com GPS "de verdade" isso não é comum. Usamos o termo GPS para smartphones porque fica mais fácil do que dizer "triangulação de antenas" e comercialmente também esse termo não cola. Então ficou assim mesmo "GPS".
Comment: O que coloquei é que não existe um "GPS real" nos smartphones. É um pouco inferior a um GPS dedicado,. E existe o sistema de localização pela rede de telefonia (triangulação)... mas enfim, não é algo trivial para a pergunta.
Comment: @cantoni Se "de longe é um problema trivial de resolver", por que você não oferece uma resposta? Certamente iria ajudar ao AP e a outras pessoas no futuro. :)
Comment: @cantoni Ah, ok. Entendi. rs Sem problemas.
Comment: @RenanCesar, sabe me dizer se o celular que você está usando aceita apenas o modo A-GPS (Assisted-GPS)?
Answer: A precisão real de um dispositivo depende do chipset, da localização onde você se encontra ( em um lugar fechado, tende se a ser mais impreciso), entre outras.
A precisão típica de um dispositivo GPS de mão seria algo como 30% das suas medidas no prazo de 50 metros da posição real (sim, os dispositivos são imprecisos).
Para tentar minimizar, você pode levar em consideração apenas as posições mais precisas!
Segue um exemplo:
var options = {
  enableHighAccuracy: true,
  timeout: 5000,
  maximumAge: 0
};

function success(pos) {
  var crd = pos.coords;
  if(crd.accuracy < 25){
      // usamos apenas as com presição menor que 25 metros
   }
};

function error(err) {
  console.warn('ERROR(' + err.code + '): ' + err.message);
};

navigator.geolocation.getCurrentPosition(success, error, options);

EDIT
enableHighAccuracy : O atributo enableHighAccuracy fornece uma dica de que o aplicativo gostaria de receber os melhores resultados possíveis. Isso pode resultar em tempos de resposta mais lentos ou no aumento do consumo de energia. O usuário também pode negar essa capacidade, ou o dispositivo pode não ser capaz de fornecer resultados mais precisos do que se o sinalizador não foi especificado. A finalidade pretendida deste atributo é permitir que os aplicativos informem a implementação de que eles não exigem correções de geolocalização de alta precisão e, portanto, a implementação pode evitar o uso de provedores de geolocalização que consumam uma quantidade significativa de energia (por exemplo, GPS). Isso é especialmente útil para aplicativos que funcionam em dispositivos alimentados por bateria, como telefones celulares.
accuracy : Este indica o nível de precisão das coordenadas de latitude e longitude. É especificado em metros e deve ser suportado por todas as implementações. O valor do atributo accuracy deve ser um número real não negativo.
No exemplo acima, pegamos apenas as coordenadas com um precisão menor que 25 metros. Isto quer dizer que ele pode estar em um raio de 25 metros desta posição.
Referência
Comment: Olá @LuizVieira, editei a resposta pra tentar sanar suas dúvidas!
Comment: Boa! Ganhou o meu +1.
Comment: Não entendi duas coisas: 1) precisão menor quer dizer precisão maior? 2) qual é o intervalo (range) da precisão (seria de 0 a 100)? Enfim, tem certeza que não tá invertendo a lógica?
Comment: Relacionado: [Gps ou triangulação de antenas - Como saber a precisão da localização?](http://pt.stackoverflow.com/q/124462/2541)@LuizVieira


Russian National Ice-Hockey Team

Question: I've seen Russia lose its match in the Olympics (2014). From the commentators on the television, I was informed that many of them play in the NHL and are millionaires.
How many do play in the NHL and are in fact millionaires?
Comment: Millionaire in terms of USD or [Russian Rubles](https://www.google.com/search?q=russian+currency+to+us+dollar&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a&channel=sb)?
Comment: @edmastermind29 didn't I mention Bitcoins?
Comment: Oh, virtual millionaires. Got it.
Answer: How many do play in the NHL?
The roster consists of 25 players total for the Russian Men's National ice hockey team. Of those 25 players, 16 of them are in the NHL. The remaining 9 play in the Russian Superleague (KHL).

How many are millionaires?
Well, I guess this is a tough question to answer. Most likely, all of them are millionaires to some degree. Certain players like Malkin, Ovechkin, etc., certainly are making way more money than most of their other teammates. But, if you take a look at the players' Wiki pages, contract information is generally given there about each one. The players in the KHL's contract information doesn't appear on their Wiki pages as often; however, I remember one of the announcers rambling on about how much money Radulov makes in the KHL ($9.2m/year), and considering the popularity of the sport in Russia, it's probably a safe bet to say all the players on their Olympic squad are millionaires.

Russian National Ice-Hockey Team


Convert a String to title case in android

Question: Before marking my question as duplicate, notice that there is a fundamental misunderstanding in that thread as to what "title case" means. In the english language, Title Case is not the same as having each word capitalized. In a title, we should

Capitalize nouns, pronouns, adjectives, verbs, adverbs, and subordinate conjunctions.
Lowercase articles (a, an, the), coordinating conjunctions, and prepositions.

Since the similar question I am pointing to is also a bit dated, I was wondering if anyone is aware of any facility in android to convert a string to a title case? Or is the only option to roll out my own class?
For some background if you care, I am building an app that has some user generated contents, in addition to expert generated contents. I want my app to look intelligent and professional. So for the EditText I enable textAutoComplete|textAutoCorrect. But in addition, I want the title to look like titles, not just strings that have been typed by random users.
Comment: Funny. In German, you should only capitalize Nouns and personal Names (first, last and middle if any). And the first Letter after the Mark (of course). As shown in this Sentence. Jokes apart, this post on codereview might be of some interest... or is it? http://codereview.stackexchange.com/questions/100832/java-port-of-titlecase
Comment: I'm afraid you'll have to write some code for this.
Answer: It is very late answer but may be can help others.
public static String changeStringCase(String s) {

final String DELIMITERS = " '-/";

StringBuilder sb = new StringBuilder();
boolean capNext = true;

for (char c : s.toCharArray()) {
    c = (capNext)
            ? Character.toUpperCase(c)
            : Character.toLowerCase(c);
    sb.append(c);
    capNext = (DELIMITERS.indexOf((int) c) >= 0);
}
return sb.toString(); }

Output will be

cHANge caSE --> Change Case
CHANGE CASE --> Change Case
change case --> Change Case

etc..


Time Series using Machine Learning models: How to deal with repeated dates and cycles

Question: I'm learning about Time Series forecasting with Machine Learning models like Neural Networks, XGBoost and others. I'm not gonna use ARIMA, SARIMA, ... So, I have the following dataset:
<code>         date     store item  sales
0       2013-01     1    1    13
1       2013-02     1    1    11
2       2013-03     1    1    14
3       2013-04     1    1    13
4       2013-05     1    1    10
...       ...      ...  ...   ...
40      2017-01     1    1    18
41      2017-02     1    1    41
42      2017-03     1    1    23
43      2017-04     1    1    67
44      2017-05     1    1    56
...       ...      ...  ...   ...
912995  2013-01     2    1    63
912996  2013-02     2    1    59
912997  2013-03     2    1    74
912998  2013-04     2    1    62
912999  2013-05     2    1    82
</code>
As you can see the dates are repeating like a cycle. This is happening because for each store[1, 10] are records about the sales at that same date interval. [2013, 2017]. The task is predict sales for the next three months. I would like to know what is the best approach to do that considering these repeated dates. I'm considering the following strategies:

Train one model for each store. But even if I do that I notice one more thing. This strategy wont work either because if the problem is this dates intervals [2013, 2017] repeating for each store I have the same problem with months and days.

Add lag features one for each of the three months, encode this dates with a function very similar to the one bellow. After that I can treat this dataset as a "regular" one and make the traditional train test split and everything else.

.
<code># not exactly this function. i would change the new features names
def encode_cyclical_atts(dataset, feat_cyclic_names):
  cyclic_aux = []
  for name in feat_cyclic_names:
    column = dataset.loc[:, name]
    max_value = column.max()
    sin_values = [math.sin((2 * math.pi * x) / max_value) for x in list(column)]
    cos_values = [math.cos((2 * math.pi * x) / max_value) for x in list(column)]
    cyclic_aux.append(sin_values)
    cyclic_aux.append(cos_values)
    feat_cyclic_encoded = np.array(cyclic_aux).transpose()

  cols_names = ['sin_WindGustDir', 'cos_WindGustDir',
                'sin_WindDir9am', 'cos_WindDir9am',
                'sin_WindDir3pm', 'cos_WindDir3pm',
                'sin_Month', 'cos_Month',
                'sin_Day', 'cos_Day']
return pd.DataFrame(data=feat_cyclic_encoded, columns=cols_names)
</code>
So what do you think about these approaches?
Comment: Can you [edit] to ask a more specific question than "What do you think?" This forum works best when users ask specific, focused questions because then answers can address exactly what you want to know about. Open-ended solicitations for thoughts or feedback may be closed as off-topic.
Answer: A few things I can think of here. If you're just trying to predict sales for a specific store, you can sum up each store's item sales and predict total sales. That's one way to do it.
If you're trying to predict specific item sales, then you could group monthly sales by store and item on a monthly basis.
In either case, you can add time series features using a package like tsfresh. This gives you a bunch of time series features for each store or store/item combo. Then you use dates for three months out or total of next three month sales as your target variable and train.
This is also a problem in time series forecasting of a top down vs bottom up approach. Predict in aggregate then predict for each item or predict for each item then predict in aggregate. Here a few links to read up on those two strategies. Hope this helps.
https://towardsdatascience.com/introduction-to-hierarchical-time-series-forecasting-part-i-88a116f2e2
https://quickbooks.intuit.com/r/growing-a-business/top-down-vs-bottom-up-which-financial-forecasting-model-works-for-you/


Simperium iOS: Issues with data retention when switching accounts or signing in to existing account

Question: In one of my Simperium powered apps, I'm currently trying to implement a clean experience for users when signing out from their sync account, signing in to a sync account with existing data, switching accounts etc.
What I want to achieve is the following (suggestions welcome, if that doesn't make any sense):

When signing out of a sync account, I don't want to delete anything locally, so the user can continue using the data (which is also important because syncing is a premium feature and the user might cancel it).
When creating a new sync account (which obviously doesn't contain any data), I want to retain the local data and sync it up to the new account (which makes sense if someone starts using the app without sync and then activates sync later).
However, when signing in to an existing sync account with existing data, I need to prevent merging hell between the local and the remote data (imagine someone switching accounts!). I think that completely overwriting the local data with the remote data is what makes most sense here.

Simperium does not seem to directly support my approach, but rather only offers to delete the data upon sign out. While trying to bend Simperium to my needs, I'm currently facing two issues/questions:

What is the preferred way to delete all local data (including all Simperium meta data, including that in user defaults)? NB: I need to do this while the user is NOT logged in to Simperium, so I can't use signOutAndRemoveLocalData:YES. I guess I need to do something like [self.buckets.allValues makeObjectsPerformSelector:@selector(deleteAllObjects)]; (which is done in the sign out completion handler), but I can't find any public interface for this. Would it be save to just delete all objects directly via CoreData, or would that leave anything in user defaults that could cause issues later, for example?
I noticed that Simperium leaves its ghost data intact after signing out. This causes massive issues when signing in to another (in my case new) account, because it doesn't treat the objects as new objects to be synced up, but rather deletes all local objects (presumably because it can't find it on the remote)!

Is this really the intended behavior? I'd expect Simperium to get rid of all sync metadata after signing out, even if the data itself is not removed. Even more so in the case of the ghost data, because this data is tied directly to the sync account - it represents the remote state of the object, and I'd argue that after signing out there is no remote state anymore (because there is no "remote").
Anyways, what's the best way for me to reset the ghost data back to the initial clean state (that is, probably something like { "key" : "...", "version" : "0" })? I could do that manually, but that feels like massively interfering with the internals of Simperium.

Sorry for this lengthy question.. Any help/pointers very much appreciated, thanks!

Update/clarification
My goal is really not so much to carry data on from one account to another. Actually, as I said, my plan was to delete all local data before signing into an exising account, which should prevent any problems like duplicates - the client just discards its local state completely and uses whatever is currently stored in the remote sync account. The only case where the retained ghost data causes me problems is this:

App currently syncing with Account A.
User signs out from Account A (using signOutAndRemoveLocalData:NO completion:).
User happily continues to use the app (perhaps for quite a while)
User creates a new sync account B (that is, one that does not contain any data on the server). Now, with the default behavior, Simperium will delete all local objects that contain ghost data with version > 0 (but will leave any objects that have been created while not signed in to any sync account, thus leaving the database in a potentially invalid state with broken relationships etc.).

I just don't think that this is the behavior that the user would expect and I'm sure it will lead to many support inqueries...
My authentication flow looks exactly like you suggested: I have my own backend that manages subscriptions and generates the auth token via the Simperium Auth API and sends it to the client. But I don't really see how this solves my problems. I completely agree that it's not a common use case that users switch sync accounts (although the situation is a bit different than for mail accounts, because you can use it without sync), and I definitely don't want to encourage users to do that. My problem is that, even if it's something that happens only rarely, it should not break the app. And I don't know how I could prevent users from doing it, because they can always freely decide to signout and sign up/in again.
What behavior would you suggest to ensure a good UX? Always delete local data upon sign out?
Answer: By design, Simperium isn't meant to carry data over from Account A to Account B. Imagine a Mail client, that would upload your Inbox upon 'Signout + Sign into a different account'.
The preferred way to reset the internal metadata is, as you pointed out, the signOutAndRemoveLocalData selector. However, the SPManagedObject's simperiumKey and ghostData are left intact. The big reason we've got for not nuking those two fields is that, if you were to sign into the very same account again, you wouldn't get any duplicate entities (the lib would just know that it's the same object).
Implementing a mechanism to reset the SPManagedObject's Simperium Metadata fields is a potential bad idea, for that reason.
Nevertheless, if you wanted to test that out, you could loop through the entities to reset, nuke the fields, and call this selector, so that the change is disregarded by the lib itself.
Furthermore, if you wanted to sign into a different account, you would probably get duplicate entities issues (that is, assuming that this second account already has objects uploaded).
I'd like to suggest a potential flow for this scenario:

On Launch, your app can hit this initializer, so that the Core Data stack is ready for work.
Once your users are ready to enable Sync'ing, you could either use this method to trigger the iOS Interface, or you could implement an API on your own backend, to wrap up the Simperium User Authentication.
This Helper API could return the authToken, which would be used by the iOS client to hit this method.
If you had to disable Sync'ing, for whatever reason, you could either:
A. Stop hitting the 'authenticate' method, or
B. Toggle this flag, as needed.

Assuming that your iOS App already has a User Authentication flow, wrapping up in your own backend the Simperium Auth call would be a great idea, because it would be crystal clear that the Sync'ing feature is 100% tied up to his own account.
I'm afraid that carry'ing on data from 'Account A' to 'Account B', although technically possible, sounds like a bad idea. As of now, the only supported mechanism to prevent duplicates relies on having the same simperiumKey locally and remotely. And (as far as i understand), that assumption won't be true in your scenarios.
Hope that helps!
Jorge
Comment: Thanks a lot for your answer, Jorge! Great to see such responsive support from you guys. I've just updated my question with a response/clarification (because its too long for a comment).
Answer: I'm sorry about the late reply!. Yes, i believe the best UX, as you say, would be to simply delete the data upon LogOff.
You can always enable or disable sync'ing (by means of the networkEnabled flag). This could help you handle the Sync'ing as a premium feature.
However, upon logout (personally speaking) i would always expect the app to nuke all of the 'personal data' i've added. Imagine how weird it'd be if Mail.app wouldn't cleanup your inbox, even after nuking an email account.
Hope you're doing great!
Jorge
Comment: Thanks for your comment. I agree that this behavior would not be expected in a mail app, but I'm not building a mail app.. ;-) In my app, the account is not a "personal" account, but rather a means to sync rather uncritical data between different family members. You can use the app without sync and "signing out" only means "signing out from the sync account". I know many iOS apps that don't nuke the local data when signing out from sync. Anyways, thanks for your help and sharing your point of view. I think I will try my luck with implementing my original approach (delete open sign **in**).
Comment: One last thing: Could you please give me a pointer for my first question from above: What's the best way to delete all local data while **not** logged in (incl. all metadata, e.g. stuff in user defaults and Application Support directory)? `signOutAndRemoveLocalData:YES` calls `deleteAllObjects` on all buckets, but I can't find any public API to get all buckets. Simply deleting the persistent store is probably a bad idea, because it leaves the Simperium data (e.g. in Application Support) in place, right?
Comment: Ahhh sorry i've missed that one!. I'm afraid that the `signOutAndRemoveLocalData:YES` method has a check that would prevent execution if the user is *not* logged in.

What you can do is simply delete the objects from CoreData, in the way yo would normally proceed, and once the context is saved, you could call the  `[_simperium resetMetadata]` helper. There is no (public) property that would expose all of the buckets, but, if needed, it can be easily added.


Unhandled error in Deferred? Using UNIX socket with Twisted

Question: I'm completely new to network programming and event driven events. However I was able to successfully implement a pub-sub scheme using a TCP connection between my machine (server) and client machines for testing (command line). However, I need to actually use a UNIX socket with Twisted.
I am getting the following error when running the code:
Unhandled error in Deferred

Here is my code for pub_sub.py:
"""
a networking implementation of PubSub using Twisted.
=============
PubSub Server
=============
A PubSub server listens for subscription requests and publish commands, and, when
published to, sends data to subscribers. All incoming and outgoing requests are
encoded in JSON.
A Subscribe request looks like this:
    {
        "command": "subscribe",
        "topic": "hello"
    }
A Publish request looks like this:
    {
        "command": "publish",
        "topic": "hello",
        "data": {
            "world": "WORLD"
        }
    }
When the server receives a Publish request, it will send the 'data' object to all
subscribers of 'topic'.
"""

import argparse
import json
import logging

from collections import defaultdict

from twisted.internet import reactor
from twisted.python import log
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import UNIXClientEndpoint, UNIXServerEndpoint, \
                                       connectProtocol
from twisted.internet.protocol import Protocol, Factory

class PubSubProtocol(Protocol):

    def __init__(self, topics):
        self.topics = topics
        self.subscribed_topic = None

    def connectionLost(self, reason):
        print("Connection lost: {}".format(reason))
        if self.subscribed_topic:
            self.topics[self.subscribed_topic].remove(self)

    def dataReceived(self, data):
        print("Data received: {}".format(data))
        try:
            request = json.loads(data)
        except ValueError:
            logging.debug("ValueError on deconding incoming data. "
                          "Data: {}".format(data), exc_info=True)
            self.transport.loseConnection()
            return

        if request['command'] == 'subscribe':
            self.handle_subscribe(request['topic'])
        elif request['command'] == 'publish':
            self.handle_publish(request['topic'], request['data'])

    def handle_subscribe(self, topic):
        print("Subscribed to topic: {}".format(topic))
        self.topics[topic].add(self)
        self.subscribed_topic = topic

    def handle_publish(self, topic, data):
        request = json.dumps(data)

        for protocol in self.topics[topic]:
            protocol.transport.write(request)
        print("Publish sent for topic: {}".format(topic))

class PubSubFactory(Factory):

    def __init__(self):
        self.topics = defaultdict(set)

    def buildProtocol(self, addr):
        return PubSubProtocol(self.topics)

class PublisherProtocol(Protocol):
    """
    Publish protocol for sending data to client, i.e. front-end web GUI.
    """
    def __init__(self, topic, **kwargs):
        self.topic = topic
        self.kwargs = kwargs

    def connectionMade(self):
        request = json.dumps({
            'command': 'publish',
            'topic': self.topic,
            'data': self.kwargs,
        })

        self.transport.write(request)
        self.transport.loseConnection()

class SubscriberProtocol(Protocol):
    """
    Subscriber protocol for client sending a request to subscribe to a specific
    topic.
    """
    def __init__(self, topic, callback):
        self.topic = topic
        self.callback = callback

    def connectionMade(self):
        request = json.dumps({
            'command': 'subscribe',
            'topic': self.topic,
        })

        self.transport.write(request)

    def dataReceived(self, data):
        kwargs = json.loads(data)

        self.callback(**kwargs)

class PubSub(object):

    def __init__(self, path='./.sock'):
        self.path = FilePath(path)
        self.reactor = reactor

    def _make_connection(self, protocol):
        endpoint = UNIXClientEndpoint(reactor, self.path)
        connection = connectProtocol(endpoint, protocol)

    def subscribe(self, topic, callback):
        """
        Subscribe 'callback' callable to 'topic'.
        """
        sub = SubscriberProtocol(topic, callback)
        self._make_connection(sub)

    def publish(self, topic, **kwargs):
        """
        Publish 'kwargs' to 'topic', calling all callables subscribed to 'topic'
        with the arguments specified in '**kwargs'.
        """
        pub = PublisherProtocol(topic, **kwargs)
        self._make_connection(pub)

    def run(self):
        """
        Convenience method to start the Twisted event loop.
        """
        self.reactor.run()

def main():
    path = FilePath("./.sock")
    endpoint = UNIXServerEndpoint(reactor, path)
    endpoint.listen(PubSubFactory())

    reactor.run()

if __name__ == '__main__':
    main()

Any help would be greatly appreciated on what I am doing wrong.
Thank you,
Brian
Answer: You appear to be running your software on Windows.  Alas, UNIX sockets are not available on Windows.  If you want to use UNIX sockets, you need to use a more POSIX-ish environment - Linux, *BSD, OS X, etc.
Comment: Thank you! I realized this last night and ran the same code on my Linux machine and everything worked fine. I appreciate your help :)


Evaluate the integral $\int_{0}^{\infty}e^{ax^3+bx^2}\,\mathrm{d}x$

Question: Is there a closed form for $$\int_{0}^{\infty}e^{ax^3+bx^2}\,\mathrm{d}x $$?
Comment: no need for a closed form because the definite integral is not convergent anyways (supposing $a$ not null).
Comment: @OJB I tried it with WolframAlpha, and even with extra computational time it still timed out, so it's unlikely that there's a simple closed form expression.
Comment: I assume $b \le 0$ as well. It will be convergent but finding a closed form is something else...
Comment: @KeshavSrinivasan So? This is clearly beyond WA. I'm 99.99%  sure a closed form exists for this.
Comment: $\displaystyle\int_0^\infty e^{-x^n}dx=\Gamma\Big(1+\tfrac1n\Big)$
Comment: One can always Taylor expand $e^{bx^2}$, integrate the expression term by term and turn the integral into a sum of 3 generalized hypergeometric functions. I think it is a sum of 3 $_2F_2$.
Comment: @OJB What makes you so sure?
Comment: @KeshavSrinivasan I found this on wiki https://upload.wikimedia.org/math/c/c/3/cc3a9fca8785f9c913757462a05cee87.png
Answer: Let $a<0$, and changing variable $y = (-a x^3)$:
$$
   \int_0^\infty \exp(a x^3 + b x^2) \mathrm{d}x = \alpha\int_0^\infty \exp\left(-y + \beta y^{2/3} \right) y^{-2/3} \mathrm{d}y
$$
where $\alpha = 1/\left(3 (-a)^{1/3}\right)$ and $\beta = b (-a)^{-2/3}$.
Now, using $y^{-2/3} \exp\left(\beta y^{2/3}\right) = \sum_{m=0}^\infty \frac{1}{m!} \beta^{m} y^{2 (m-1)/3}$, and interchanging the order summation and integration as warranted by Tonelli theorem:
$$
 \int_0^\infty \exp\left(-y + \beta y^{2/3} \right) y^{-2/3} \mathrm{d}y = \sum_{m=0}^\infty \frac{\beta^m}{m!} \int_0^\infty \mathrm{e}^{-y} y^{2 (m-1)/3} \mathrm{d}y = \sum_{m=0}^\infty \frac{\beta^m}{m!} \Gamma\left(\frac{2m}{3} + \frac{1}{3}\right)
$$
Using duplication and triplication formulae for $\Gamma$-function
$$
    \Gamma(2x) = \frac{2^{2x}}{2 \sqrt{\pi}} \Gamma(x) \Gamma\left(x+\tfrac{1}{2}\right) \quad \Gamma(3x) = \frac{3^{3x}}{2 \sqrt{3} \pi} \Gamma(x)  \Gamma\left(x+\tfrac{1}{3}\right)   \Gamma\left(x+\tfrac{2}{3}\right)
$$
we can rewrite the summand:
$$
   \frac{\Gamma\left(\frac{2m}{3} + \frac{1}{3}\right)}{m!}  =  \frac{\Gamma\left(\frac{2m}{3} + \frac{1}{3}\right)}{\Gamma(m+1)}  = \frac{2^{1/3} \sqrt{\pi}}{\sqrt{3}} \left(\frac{2^{2/3}}{3}\right)^m \frac{\Gamma\left(\frac{m}{3} + \frac{1}{6}\right)}{\Gamma\left(\frac{m}{3} + 1\right) \Gamma\left(\frac{m}{3} + \frac{1}{3}\right) }
$$
It now suffices to split the sum
$$\sum_{m=0}^\infty g(m) = \sum_{k=0}^\infty g(3k) + \sum_{k=0}^\infty g(3k+1) + \sum_{k=0}^\infty g(3k+2)
$$
and write the answer in terms of hypergeometric functions:
$$
\begin{eqnarray}
  \int_0^\infty \exp(a x^3 + b x^2) \mathrm{d}x &=& \tilde{\alpha} \frac{\Gamma(1/6)}{\Gamma(1/3)} \cdot {}_1F_1\left(\tfrac{1}{6}; \tfrac{1}{3}; \left(2^{2/3} \frac{\beta}{3}\right)^3 \right) \\
 &+& \tilde{\alpha} \cdot  \frac{2^{2/3} \beta}{3} \cdot \frac{\Gamma(1/2)}{\Gamma(2/3)\Gamma(4/3)} \cdot {}_2F_2\left(1, \tfrac{1}{2}; \tfrac{2}{3},\tfrac{4}{3}; \left(2^{2/3} \frac{\beta}{3}\right)^3 \right) \\
 &+& \tilde{\alpha} \cdot  \left(\frac{2^{2/3} \beta}{3}\right)^2 \cdot \frac{\Gamma(5/6)}{\Gamma(1)\Gamma(5/3)} \cdot {}_1F_1\left(\tfrac{5}{6}; \tfrac{5}{3}; \left(2^{2/3} \frac{\beta}{3}\right)^3 \right)
\end{eqnarray}
$$
where $\tilde{\alpha} = \frac{2^{1/3} \sqrt{\pi}}{\sqrt{3}}\alpha $
Comment: Thanks! This is very helpful.
Answer: Here is what WolframAlpha gives me, just for $a=-1$:
$$\begin{align*}
\int^\infty_0 e^{-x^3+bx^2}\,dx=-\frac1{27}b\,\left(2\sqrt{3}\pi e^{\tfrac{2b^3}{27}}\left(I_{-\frac13}\left(-\dfrac{2b^3}{27}\right)+I_{\frac13}\left(-\dfrac{2b^3}{27}\right)\right)-9_2F_2\left(\dfrac{1}{2},1;\dfrac23,\dfrac43;\dfrac{4b^3}{27}\right)\right)\,\,\,\\\color{grey}{\text{ for }}\,\operatorname{Re}(b)<0
\end{align*}
$$
It's expressed using Bessel functions and hypergeometric functions, and WolframAlpha was just barely able to compute it with extra computational time. So just imagine how complicated an expression you'd have in the general case.
Comment: A simple substitution, with $x=a^{1/3}y$, and a different $b$, will give the general case


No Data In Array

Question: I am creating an app which uses Core Data. I have a flip-side in which the user creates the records which are to be used in the main-view. I fetch the Core Data in my ViewWillAppear in the MainViewController. The problem arises when I run the app on a new device, it returns the error that there is no data in the array.
I populate the array as follows:
<code>/*
 Fetch existing events.
 Create a fetch request; find the Event entity and assign it to the request; add a sort descriptor; then execute the fetch.
 */
NSFetchRequest *request = [[NSFetchRequest alloc] init];
NSEntityDescription *entity = [NSEntityDescription entityForName:@"Child" inManagedObjectContext:_managedObjectContext];
[request setEntity:entity];

// Order the events by creation date, most recent first.
NSSortDescriptor *nameDescriptor = [[NSSortDescriptor alloc] initWithKey:@"name" ascending:NO];
NSSortDescriptor *prizeDescriptor = [[NSSortDescriptor alloc] initWithKey:@"prize" ascending:NO];
NSSortDescriptor *neededDescriptor = [[NSSortDescriptor alloc] initWithKey:@"marblesneeded" ascending:NO];
NSSortDescriptor *colorDescriptor = [[NSSortDescriptor alloc] initWithKey:@"color" ascending:NO];

NSArray *sortDescriptors = [[NSArray alloc] initWithObjects:nameDescriptor,prizeDescriptor,neededDescriptor,colorDescriptor, nil];
[request setSortDescriptors:sortDescriptors];
[nameDescriptor release];
[colorDescriptor release];
[prizeDescriptor release];
[neededDescriptor release];
[sortDescriptors release];

// Execute the fetch -- create a copy of the result.
NSError *error = nil;
records = [[self.managedObjectContext executeFetchRequest:request error:&error] retain];

if ([records count] == 0) {
    recordsempty = YES;
}

[request release];
</code>
And I get the following error:
<code>-[__NSArrayI objectAtIndex:]: index 0 beyond bounds for empty array'
</code>
Does anyone have any ideas on helping prevent this crash when a user installs my app from the App Store?
Thanks
Comment: it would be helpful if you gave details about the crash as well as some code that shows how you are putting objects into the array.
Comment: @timthetoolman, I have added the code and the errors.
Comment: You should indicate where in the code you are getting the error.  If you only get the error when there's no data, then somewhere you have some code that expects there to be data at index:0.
Comment: @OwenHartnett, Yes, if The User has not entered any data, the fetch will not return any data causing the error.
Comment: Yes, but where does the error happen in your code? Somewhere your code is acting like there is data there and is trying to get index:0 and it isn't there.
Comment: @OwenHartnett, The Default for pageControl.CurrentPage is 0.
Answer: The Problem was fixed after I added an if statement to not try to display anything if the array is empty. Then, I used an UIAlertView to direct the user to the flipside.

No Data In Array


Show that if the eigenvalues of a real matrix are not real, then the matrix cannot be symmetric

Question: Question for the problem
I have not been able to make any progress in this problem and would appreciate any help if possible.  I have been able to prove that eigenvalues of a symmetric matrix are real.
Comment: Then you're done (by logic). (You just said) If the matrix is symmetric then it's eigenvalues are real. Let's suppose your matrix is symmetric. Then it's eigenvalues are real. Contradiction!
Comment: Proof by contraposition is totally fine, mathematically @UMass1234
Comment: Yeah logically I can deduce it, but i think I have to find a way to show that the above holds.
Comment: Do you know how to form the contrapositive of "If A then B"?  What is the contrapositive of "If A is a symmetric matrix, the eigenvalues of A are all real"?
Comment: Proof by any means is fine. So, in particular, @AhmedS.Attaalla's suggestion is perfectly good.
Answer: Suppose $Av = \lambda v$ with $\lambda $ complex and $A$ real.
Then $x^* Ax = x^* (\lambda x) = \lambda \|x\|^2$
and $(A x)^* x = x^* A^T x= \overline{\lambda} \|x\|^2$.
Hence $x^* (A -A^T) x = (\lambda - \overline{\lambda}) \|x\|^2 \neq 0$.
In particular, we must have $A \ne A^T$.
Comment: Thank you for the response.  I understood the idea of using contrapositive but, I was assuming my professor was looking for something shown more directly.  Thank you again.


python list that matches everything

Question: I probably didn't ask correctly: I would like a list value that can match any list: the "inverse" of (None,)
but even with (None,) it will match item as None (which I don't want)
The point is I have a function working with: [x for x in my_list if x[field] not in filter_list]
and I would like to filter everything or nothing without making tests like:
if filter_list==(None,): return [] and if filter_list==('*',): return my_list
PS: I wanted to simplify my question leading to some errors (list identifier) or stupid thing [x for x in x] ;)

Hi,
I need to do some filtering with list comprehension in python.
if I do something like that:
[x for x in list if x in (None,)]

I get rid of all values, which is fine
but I would like to have the same thing to match everything
I can do something like:
[x for x in list if x not in (None,)]

but it won't be homogeneous with the rest
I tried some things but for example (True,) matches only  1
Note than the values to filter are numeric, but if you have something generic (like (None,) to match nothing), it would be great
Thanks
Louis
Comment: `list` is a reserved word, you shouldn't use it as an identifier.
Comment: indeed: this was for demonstration purpose only:I removed it anyway
Comment: may be you could post an example of what you want? say you have `my_list = [1, 2, None, 4]` and `filter_list` = [2]` what should be the output. do add inputs of your own
Comment: @Louis: Please delete the old question; it makes no sense.  It's actually better to delete this question and start again with a brand new question.
Comment: Note that `(None,)` is not exactly a match-nothing: it does math `None`.  What you want to match nothing is `()`.
Answer: __contains__ is the magic method that checks if something is in a sequence:
class everything(object):
    def __contains__(self, _):
        return True

for x in (1,2,3):
    print x in everything()

Answer: The better syntax would be:
[x for x in lst if x is None]
[x for x in lst if x is not None]

Answer: What do you mean by

I would like to have the same thing to match everything

Just do
[x for x in list]

and every item in list is matched.
Answer: You could change your program to accept a filter object, instead of a list.
The abstract base filter would have a matches method, that returns true if x *matches".
Your general case filters would be constructed with a list argument, and would filter on membership of the list - the matches function would search the list and return true if the argument was in the list.
You could also have two special subclasses of the filter object : none and all.
These would have special match functions which either always return true (all) or false (none).
Answer: You don't need an if, you can just say
[x for x in list]

Comment: How can you justify this:
>>> l = [None, 1, 3]
>>> [x for x in l]
[None, 1, 3]
Comment: @aatifh: that's not python syntax
Comment: the `if` was because I filter on a field: ti's now correct in question
Answer:
but I would like to have the same
  thing to match everything

To match everything, you don't need if statement
[x for x in list1]

or If you really like to do
[x for x in list1 if x in [x]]

Answer: Answering your revised question: the list that "matches" all possible values is effectively of infinite length. So you can't do what you want to do without an if test. I suggest that your arg should be either a list or one of two values representing the "all" and "none" cases:
FILTER_NONE = object() # or []
FILTER_ALL = object()

def filter_func(alist, filter_list):
    if filter_list is FILTER_ALL:
        return []
    elif filter_list is FILTER_NONE:
        return alist
        # or maybe alist[:] # copy the list
    return [x for x in alist if x not in filter_list]

If filter_list is large, you may wish the replace the last line by:
    filter_set = set(filter_list)
    return [x for x in alist if x not in filter_set]

Alternatively, don't bother; just document that filter_list (renamed as filter_collection) can be anything that supports  __contains__() and remind readers that sets will be faster than lists.

python list that matches everything


Scale Invariant Statistical Distances

Question: Problem
Suppose we have empirical distributions to two $n$-dimensional random variables $X = (X_1, X_2, ..., X_n)$ and $Y = (Y_1, Y_2, ..., Y_n)$. The goal is to find $k < n$ components, such that the $1$-dimensional distributions of $X_{i_1}, X_{i_2}, ..., X_{i_k}$ and $Y_{i_1}, Y_{i_2}, ..., Y_{i_k}$ are "most similar" to each other, i.e. $\text{argmin}_i dist(X_i, Y_i)$.
We compare $X_1$ to $Y_1$, $X_2$ to $Y_2$ etc.
As an added difficulty due to physical units, the scale (max, min, std) of $X_i$ might be very differnt from the scale of $X_j$ for any $i \neq j$. But $X_i$ and $Y_i$ are approximately comparable (think same measurement device and test case but slightly different product).
Question: What statistical distances are available that capture similarity between the components of $X$ and $Y$, are scale invariant and emphasize tail probabilities?
Our current aproach is as follows:
For each $i=1,...,n$

Take the Union $U_i$ over all samples from $X_i$ and $Y_i$
Compute $m = mean(U_i), s = std(U_i)$
$X_i' = (X_i - m) / s, \, Y_i' = (Y_i - m) / s$
$d(X_i, Y_i) = \int_{\mathbb{R}} |p_i(x) - q_i(x)| |x|^p dx$ where $p$ denotes the PDF of $X'_i$ and  and $q_i$ denotes the PDF of $Y'_i$.

The factor $|x|^p$, ($p=5$) stems from the fact, that we don't really care about density around $0$.
Comment: did you try K-S stat?
Comment: Maybe that's where you should start. It's the common way to measure distance between distributions and its scale invariant
Comment: Unfortunately, the KS statistic does not focus on tail behavior and looks difficult to modify to make it so.  This is why other statistics, such as the [Anderson-Darling](https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test) and [Shapiro-Wilk](https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test) statistics, have been invented.  cc @Aksakal
Comment: @whuber right, so similar stats like AD should be a good place to start
Comment: Could you clarify what you mean by "counterparts"?
Comment: Ammended the sentence. For which i, Xi and Yi are the closest.
Comment: @Aksakal not yet.
Answer: There's probably a better answer incoming but here's something to get you started. It looks like what you're using is related to the Total Variation Distance
$$
\delta(P,Q) = \frac12 \int_{\mathbb{R}}|P(x) - Q(x)|\,dx
$$
(see this question for why this is the correct formula). This is unit independent since $P(x)$ is in units of $\frac{1}{x}$ and $dx$ is in units of $x$. It's also bounded between 0 and 1, making it a reasonable things to use to compare distribution over spaces with different units. The addition of the $x^p$ however will add in units and make comparison between the components $i$ and $j$ difficult.
If you want to keep the $x^p$ you could form a sort of "almost" $L^p$ metric using
$$
\delta(P,Q) = \left(\int_{\mathbb{R}}|P(x) - Q(x)|^px^{p-1}\,dx\right)^{\frac1p}
$$
This will tend to emphasize values away from 0 more, but I would hesitate to say more about the behavior of the metric, or how to metric will change for different distribution, you could do some computational investigations though.
You can find a long list of metrics here but no others jumped out at me as scale invariant.
Comment: Basically assume x is in units of meters $m$ as an example (in general an engineer of physicist might denote the units by [x]). It's not clear how to make sense of expressions with different units, is $10\,m<15\, s$ for example. As such, we only want to compare unitless quantities. In general, if $x$ has units $m$, than $x^2$ has units $m^2$, etc. A PDF $P(x)$ has units of 1/[x], since it takes an interval in [x] and returns a probability, which is unitless. $dx$ has units $[x]$, so $[\delta] = \frac{1}{[P(X)]}[x^p][dx] = [1]$ is unitless
Comment: It seems like you guys have put some good thinking into this, I would note that things being unitless doesn't mean they're on the same scale but for total variation that is fixed by the probability bound. So here, I'm just trying to create you another unitless quantity, I think it should work once you scale as you've done, but I'm not at all sure. Interesting question though!
Comment: Thanks, we already did some investigation into TV by setting p to 0 and that worked fine, although it would diverge from how engineering is using these densities. What exactly does x^(p-1) here?
Comment: Thanks, I found the problem interesting, too. It seems developing a sensible metric is a more difficult problem than we originally thought.
Comment: The rescaling before calculating the metric does not work in principle though, since we first scale both time series essentially based on their tails, then we estimate the size of exactly these rescaled tails. That's just circular logic.
Comment: I do think your rescaling is "correct" though, because you want to center your data and put it on the same scale for comparison. Can you generate a bunch of synthetic data in different situations and classify how things behave? It would be an interesting exploration of parameter space.

Scale Invariant Statistical Distances


$2\int_{0}^{1} \cos(\pi x)\sin(m\pi x)dx = \frac{2m(\cos(m\pi)+1)}{\pi (m^2-1)}$ when $m \ge 2 $

Question: $$2\int_{0}^{1} \cos(\pi x)\sin(m\pi x)dx = \frac{2m(\cos(m\pi)+1)}{\pi (m^2-1)},\; m\ge2$$
I can't seem to be able to prove this.
I have tried using the identity $\sin(a)\cos(b) = 1/2 (\sin(a-b)+\sin(a+b))$ to get  $$\int_{0}^{1} \cos(\pi x)\sin(m\pi x)dx = \frac{1}{2} \int_{0}^{1} \sin((m-1)\pi x) + \sin((m+1)\pi x) dx$$ and then solving but am unable to reproduce the solution.
Comment: What have you tried?
Comment: Hint: linearize $\cos a\sin b$.
Comment: This question is missing context or other details: Please provide additional context, which ideally explains why the question is relevant to you and our community. Some forms of context include: background and motivation, relevant definitions, source, possible strategies, your current progress, why the question is interesting or important, etc.
Comment: ive tried using the identity sin(a)cos(b)=1/2(sin(a−b)+sin(a+b)) but have had no success
Comment: @pablo_mathscobar that's exactly the right idea.  So where are you getting stuck?
Comment: im confused as to how it simplifies to the form on the RHS
Answer: Your method gives $\int_0^1 (\sin (m + 1)\pi x+\sin (m -1)\pi x)dx=\frac{-1}{\pi}\bigg(\frac{\cos (m+1)\pi -1}{m+1}+\frac{\cos (m-1)\pi -1}{m-1}\bigg)$. As the cosines are both $-\cos m\pi$, this simplifies to $\frac{2m (\cos m\pi +1)}{\pi (m^2-1)}$. As a sanity check, let's take $m=2$, so my claim is that the answer is $\frac{8}{3\pi}$. Indeed $$2\int_0^1\cos \pi x\sin 2\pi x dx=2\int_0^1 2\sin \pi x\cos^2 \pi x dx,$$which by $u=\cos\pi x$ gives $$\int_{-1}^1\frac{4}{\pi}u^2 du=\frac{8}{3\pi}.$$
Comment: apologies i forgot to write the 2 when typing the equation out my confusion comes from where you say that the cosines are both $-cos(m\pi)$? how does that simplify
Comment: @pablo_mathscobar From the identity $\cos (y+\pi)=-\cos y$.
Comment: thank you!!!!!!!


Call to undefined function flag_get_flag() with Flag 4.x

Question: I'm using the 8.x-4.0-beta3 release of the Flag module in a custom D9 module to flag a newly programmatically created node using a flag I created in the UI.
<code>$flag = flag_get_flag('new_call');
$flag->flag('flag', $call_node->nid);
$call_node->save();
</code>
But this results in this error when it is called:
<code>Error: Call to undefined function Drupal\zoom_call\flag_get_flag() in Drupal\zoom_call\ZoomApiWebhookEventSubscriber->clientcall() (line 75 of /var/www/html/modules/zoom_call/src/ZoomApiWebhookEventSubscriber.php)
</code>
Does someone know a solution to this? Is flag_get_flag() deprecated in D9 perhaps?
Comment: Try \flag_get_flag
Comment: @cilefen Gives the same undefined function error.
Answer: You need to use the flag service:
<code>$flag_id = 'bookmark';

$flag_service = \Drupal::service('flag');
$flag = $flag_service->getFlagById($flag_id);

// Flag an entity with a specific flag.
$flag_service->flag($flag, $entity);

// Unflag an entity with a specific flag.
$flag_service->unflag($flag, $entity);
</code>
Documentation for Flag 4.x is lacking but I made a list of the basics.
Comment: That works, thank you, for both the solution and list.