When does one use an extra stave for piano music?

Question: In complex piano music, one sometimes notates the notes over an additional stave - though I cannot say for sure I always know the reason. Here's an example from Stravinsky's Firebird, arranged by Agosti:

I myself cobbled together an arrangement of the finale of Shosty's 11th symphony for solo piano. The ending features a loud melody across the register and dominant percussion underneath, which I transcribed like this:

I elected to put the percussion on a separate staff, even though it is also played by the left hand, because:

it is of a very different character from the chords
it is pseudo-optional - you could add any actual percussion instrument to replace this
I personally found this more readable

But, as I am not actually schooled in engraving (I do this just for fun), chances are not everyone will agree with me.
Is this a proper use of a separate stave? If so, or if not, when does one add additional staves to piano music? What factors does one consider?
Comment: Just for clarity, are you asking exclusively about piano _arrangements?_ Some piano music - Debussy comes to mind - uses three staves in the _original piano music._
Comment: On the other hand the first two bars of the treble staves have different expression marks. I suppose if they were on one staff you would need more markings to make clear the alternating expressions. Again, the origin of this may be the original orchestration.
Comment: @MichaelCurtis Piano music in general. To be honest, I'm not sure why one would have different notation rules for arrangements than for original compositions, if that's what you are implying.
Comment: I'm reasonably sure that the Stravinsky example uses the third stave -- and the choice of connected eighth notes - to specify which hand should be playing which set of notes.
Comment: @CarlWitthoft that's possible (though it looks awkward to play). I also considered that it is because the middle stave is at a different volume from the higher one, at least for the first two bars.
Comment: @KeizerHarm, in Stravinsky example I couldn't see an immediate reason for the two treble staffs - it seems to make the middle staff look like it goes to much lower notes but it actually doesn't - then I thought the additional staff may represent some separation in instrumentation.
Answer: Elaine Gould recommends the use of additional staves "if multiple parts are otherwise hard to read" (Behind Bars, p. 332). That is clearly the case here, since with only one bass stave, chords would collide with the pauses and tuplet braces. Also, the staves should be arranged in pitch order, and a given stave should consistently be assigned to the left or right hand. So your arrangement is a textbook example.
However, you should probably extend the brace to all three staves, as in the Agosti example. As it is, this looks like organ music.
Answer: There isn't much reason for three staves in the second example. It is readable enough on two.

The first one is more arguable. The first two parts are probably better on three staves though it isn't really necessary, but the in the last bar I personally detest the style of using cross-staff notes just to save a few leger lines, and IMO those bars are more readable on two staves than three.
I don't understand the comments about "different dynamics" here. The first note is marked with an accent (sfff). The following passage is marked "quasi f". If the "quasi f" was meant to apply to the middle staff only, it should have been below the middle staff, not where it is. (And let's not start a debate about what "quasi f" is supposed to mean. Either you play a note loud or you don't!)

Of course there are times when you really do need 3 (or more) staves (e.g. from Scriabin Sonata No. 7)
Comment: Could you explain why more staves *are* needed in the Scriabin example? I have an idea but I'd want to know what makes the difference to you.
Comment: The top two staves should be combined into one, at least given what this excerpt shows.  Debussy has written even denser passages in only two staves.
Comment: I disagree with the assertion that the "quasi f" applies to the top staff because it's not below the middle staff. In this case, it's written _on top of_ a staff, so that's almost certainly the staff to which it applies.
Comment: @CamilleGoudeseune That's what I thought. Maybe the right hand plays the 1st and 3rd staves, and the left hand the 2nd and 4th staves?
Comment: @Keizerharm good point, RH+LH 3&4, then R+L 1&2, argues for "two pairs" of staves.
Answer: I just wanted to address a point that came up in comments, namely, "I'm not sure why one would have different notation rules for arrangements than for original compositions."
The response is that sometimes an arranger is also trying to create a sort of reduction when writing out a piano version of a piece for orchestra or other large ensemble.  I've even seen some piano arrangements that contain notations about what instrument was originally playing a particular voice.  That can be helpful sometimes in gauging things like articulation or trying to better emulate the effect of various instruments.  (This can blur the line between a "short score" which is often a reduction of orchestral parts written on 3-4 staves, and a true "arrangement" which is necessarily practically playable on piano with only two hands.  Sometimes orchestral transcriptions do a bit of both.)
In that case, sometimes additional staves are used to separate out instrumental parts.  I would have to look at the score of Firebird to know whether this is at all applicable in the Stravinsky example, but an arranger might use multiple staves in a situation like this to emphasize that different instruments are playing syncopated parts, which would likely give a more disjoint effect when performed by an orchestra.  Seeing the notation like this may make it more clear to a pianist that the separate staves should be seen/heard as separate parts normally played by separate instruments and therefore should be articulated as such.
Comment: @KeizerHarm: Regarding your own example, perhaps.  But as seen in the other answer where someone actually typeset your example in two staves, it's very clear that the bass rhythm is its own voice with its own rhythm, whereas in the Stravinsky case, the voices are in the same register, overlapping in rhythm, etc. so it's harder to see them as two separate elements within the texture without separating them further.  In the end, it's a judgment call, but I don't personally think it's necessary in your example.
Comment: Thank you for that insight. I suppose that's an argument in favour of having three staves in my example, as the lower one is very much its own instrument (the percussion). Though I may need to write that explicitly in a comment.


How to change date format to custom in Views? (drupal 6, views2)

Question: I have a view that includes a CCK date/time field. It automatically displays the date like this if I choose Long as the format:
Tuesday, April 17, 2012 - 13:30
I need it to display in 12-hour time, not 24-hour, but the view field config doesn't allow me to enter a custom format for my date (for the label, yes - the actual value, no). How can I override the date formatting for just this one view?

EDIT
Never mind; I found a way to do it here.
Answer: Answer found here. Basically, you create a new date format in admin that is then available sitewide in any date format dropdown.


Is there a better way to format these elements_by_id?

Question: I have the following code I'm repeating and wondered if anyone had some advise as to how to write this more efficiently:
def get_description(links):
    for link in links:
        description = driver.find_elements_by_id('some-id')
        description = [x.text for x in description]
        description = " ".join(description)
        title = driver.find_elements_by_id('different-id')
        title = [x.text for x in title]
        title = " ".join(title)
        company = driver.find_elements_by_id('another-different-id')
        company = [x.text for x in company]
        company = " ".join(company)
        location = driver.find_elements_by_id('location-id')
        location = [x.text for x in location]
        location = " ".join(location)+ " United Kingdom"
        salary = driver.find_elements_by_xpath("//*[@id='randomly generated id']/div[3]/span[1]")
        salary = [x.text for x in salary]
        salary = " ".join(salary)

I tried defining a separate function called 'element_parse' as follows:
def parse_element(x)
    x = [y.text for y in x]
    x = " ".join(x)

then calling this the main function by doing:
description = driver.find_elements_by_id('some-id')
parse_element(description)

But alas! No joy.
Not a show stopper as I've got it working but feel like there's a lot of repetition in here I want to clean up!
Comment: If you want to improve working code, ask your question on [CodeReview](https://codereview.stackexchange.com/)
Comment: Thanks Andersson - I wasn't aware of this. I'll have good use for CodeReview so appreciate you bringing this to my attention!
Answer: You're almost there. You need to return the value of x from your function, and reassign the value to that result. So:
def parse_element(x)
    x = [y.text for y in x]
    x = " ".join(x)
    return x

...
description = driver.find_elements_by_id('some-id')
description = parse_element(description)

Comment: Thanks Daniel! This definitely helped. I finished up by passing in another variable ('y') to the function which holds the id.
Answer: You can shake off the repitition by trying something like below. Moreover, you don't need to create another function to clean things up.
def get_description(links):
    for link in links:
        description = ' '.join([x.text for x in driver.find_elements_by_id('some-id')])
        title = ' '.join([x.text for x in driver.find_elements_by_id('different-id')])
        company = ' '.join([x.text for x in driver.find_elements_by_id('another-different-id')])
        location = ' '.join([x.text for x in driver.find_elements_by_id('location-id')])
        salary = ' '.join([x.text for x in driver.find_elements_by_xpath("//*[@id='randomly generated id']/div[3]/span[1]")])


Ninject rebinding a method

Question: Using Ninject i would like to rebind one method to another implementation is that possible ?
I will elaborate, I have this interface with two different implementations:
public interface IPersonFacade
{
  List<string> GetPeople();
  string GetName();
}

public class PersonFacade:IPersonFacade
{
  //Implement Interface fetching data from a db.
}

public class PersonFacadeStub:IPersonFacade
{
  //Implement Interface with some static data
}

I am using the Ninject mvc extension and have my NinjectModule implementation:
public class ServiceModule:NinjectModule
{
  public override void Load()
  {
    Bind<IPersonFacade>().To<PersonFacade>();
  }
}

So back to my question, is it possible to rebind  the method GetPeople() so it uses the implementation from the PersonFacadeStub but IPersonFacade keeps using the GetName from the PersonFacade ?
Answer: I don't think it is possible. NInject like any other DI container manages types, not methods. If you want to use different types for different methods of the same interface pattern Composite may help:
public class CompositePersonFacade : IPersonFacade
{
    private readonly IPersonFacade realFacade;
    private readonly IPersonFacade stubFacade;

    public CompositePersonFacade(IPersonFacade realFacade, IPersonFacade stubFacade)
    {
        this.realFacade = realFacade;
        this.stubFacade = stubFacade;
    }

    public List<string> GetPeople()
    {
        return stubFacade.GetPeople();
    }

    public string GetName()
    {
        return realFacade.GetName();
    }
}

and also modify binding:
Bind<IPersonFacade>().To<CompositePersonFacade>()
    .WithConstructorArgument("realFacade",
                  context => context.Kernel.Get<PersonFacade>())
    .WithConstructorArgument("stubFacade",
                  context => context.Kernel.Get<PersonFacadeStub>());

Comment: +1 Nice. @Nizolai Have a look whether Ninject.Extensions.Interception can be of any use (depends on whether you want literally what you're asking for...)

Ninject rebinding a method


algorithms to pick best collection of classifiers

Question: I have a bunch of methods that classify a binary outcome. I'm trying to figure out if some combination of those classifiers is better than any others.
I'm hoping to run a bunch of methods. I've thought of running random forests/cart and perhaps a SVM model.
What are some other common methods typically done for this?
Answer: You're asking about whats known as 'ensembling'. I'd suggest reading about it here http://en.wikipedia.org/wiki/Ensemble_learning. Two popular methods are boosting and bagging.


theoretical question regarding deduction and relation between $\vdash$ and $\vDash$

Question: i have a very basic and theoritical question to help me understand the basics of logics and deduction systems. trying to understand the basics between the difference and deduction of $ \vdash$ $\vDash$ and the relation and deduction between them.
D is an deduction system of proposition calculus. it's axioms are all the propositions that are not tautologies and it's only deduction rule is: $\frac{a \lor b}{a \land b}$.
1)does it apply for every set of proposition x, so that $x \vdash a \Rightarrow x \vDash a$?
2)does $x \not \vDash a \Rightarrow x \vdash a$ ?
thank you very much for aiding me learn.
Comment: See posts [Why are $\vdash$ and $\vDash$ symbols from metalanguage](https://math.stackexchange.com/questions/1218354/why-are-vdash-and-vdash-symbols-from-metalanguage) and [Meaning of symbols $\vdash$  and $\models$](https://math.stackexchange.com/questions/1115066/meaning-of-symbols-vdash-and-models).
Answer: You have to prove a sort of soundness theorem for your calculus.
Hint about soundness :

if $\Gamma \vdash a$, the $\Gamma \vDash a$.

Assume that $\Gamma \vdash a$ and consider the ususal cases :
(i) $a \in \Gamma$; then obviously $\Gamma \vDash a$, because every assignment $v$ that satisfies every formula in $\Gamma$ will satisfies also $a$.
(ii) $a$ is an axiom; this means that $a$ is not a tautology. Consider the case $a := p$.
According to the definition of derivation, we have $\{ q \lor r \} \vdash p$, but obviously : $\{ q \lor r \} \nvDash p$.
(iii) $a$ is derived with the inference rule from a previous formula in the derivation.


Storing a million images in the filesystem

Question: I have a project that will generate a huge number of images.  Around 1,000,000 for start. They are not large images so I will store them all on one machine at start.
How do you recommended on storing these images efficiently?  (NTFS file system currently)
I am considering a naming scheme... for start all the images will have an incremental name from 1 up
I hope this will help me sort them later if needed, and throw them in different folders.

what would be a better naming scheme:

a/b/c/0 ... z/z/z/999

or

a/b/c/000 ... z/z/z/999

any idea on this ?
Comment: Is this you? http://i46.tinypic.com/1z55k7q.jpg
Comment: Are they tied to specific users or just generic?  Are they grouped in any fashion?
Comment: :)) yea... 1 mil. porn images :))
Comment: only generic. a bunch of images generated by some technical equipment. i am naming them incremental from 1 up just to have am idea of a time refence.
Comment: how are they going to be used/accessed?  through a bespoke app or what?
Comment: i am storing in a database the Path of each image, when a users wants to access it the db gives me the path and then i load it from there.
Comment: what about limitations regarding the number of files stored in a single folder and that stuff ?
Comment: Out of curiosity, is there any compelling reason to use NTFS rather than FAT16 or 32, in terms of performance? I am purely interested in knowing if you have run any benchmarking over various file systems to get the best.
Comment: no performance tests done. i just wanted to get this app going and this is what i had in hand.
Comment: Are you planning to delete lot of images? If so, is there a some pattern e.g. delete always oldest images, delete images randomly?
Comment: At this time there is no plan to delete files, ever.
Deleting older files randomly to keep folders clean... i doubt there is an even distribution of this thing we call "random"
Comment: how do you intend to know how to locate a particular file?
Comment: like someone mentions below. i'll just recreate the filepath from the file name. look at "Juha Syrjälä" answers
Comment: @Mike, my point was actually that you create a sequence number for image (with database sequence or autoincrement column, for example) and then create filename and path from that sequence number.
Comment: oh i definitely advocate that idea (used similar in the past). if you do groupings of letters per directory branch, just be aware of density distribution and conditions for how many 'dangling' characters to use for the actual filename - and how to handle odd versus even length names.
Comment: Check out this question: http://stackoverflow.com/questions/1257415/best-way-to-store-retrieve-millions-of-files-when-their-meta-data-is-in-a-sql-dat  You may gain some additional ideas over there...
Comment: @Juha - the filenames are already in sequence and unique
Comment: File systems are not generally a good storage mechanism for millions of small files as they tend to take up whole blocks of space (not just their true size.)  If space becomes an issue AND if access will be infrequent and in batches, you may want to consider grouping a few hundred or thousand at a time into single files (.tar or .zip)
Comment: @user28770 FAT will be a bad choice here. Since the file allocation table is linear, you have to search all possible entries to get a file. You also don't have an allocation bitmap which will result in significant fragmentation
Answer: I'd recommend using a regular file system instead of databases. Using file system is easier than a database, you can use normal tools to access files, file systems are designed for this kind of usage etc. NTFS should work just fine as a storage system.
Do not store the actual path to database. Better to store the image's sequence number to database and have function that can generate path from the sequence number. e.g:
 File path = generatePathFromSequenceNumber(sequenceNumber);

It is easier to handle if you need to change directory structure some how. Maybe you need to move the images to different location, maybe you run out of space and you start storing some of the images on the disk A and some on the disk B etc. It is easier to change one function than to change paths in database.
I would use this kind of algorithm for generating the directory structure:

First pad you sequence number with leading zeroes until you have at least 12 digit string. This is the name for your file. You may want to add a suffix:

12345 -> 000000012345.jpg

Then split the string to 2 or 3 character blocks where each block denotes a directory level. Have a fixed number of directory levels (for example 3):

000000012345 -> 000/000/012

Store the file to under generated directory:

Thus the full path and file filename for file with sequence id 123 is  000/000/012/00000000012345.jpg
For file with sequence id 12345678901234 the path would be 123/456/789/12345678901234.jpg

Some things to consider about directory structures and file storage:

Above algorithm gives you a system where every leaf directory has maximum of 1000 files (if you have less that total of 1 000 000 000 000 files)
There may be limits how many files and subdirectories a directory can contain, for example ext3 files system on Linux has a limit of 31998 sub-directories per one directory.
Normal tools (WinZip, Windows Explorer, command line, bash shell, etc.) may not work very well if you have large number of files per directory (> 1000)
Directory structure itself will take some disk space, so you'll do not want too many directories.
With above structure you can always find the correct path for the image file by just looking at the filename, if you happen to mess up your directory structures.
If you need to access files from several machines, consider sharing the files via a network file system.
The above directory structure will not work if you delete a lot of files. It leaves "holes" in directory structure. But since you are not deleting any files it should be ok.
Comment: Using a hash (such as MD5) as the name of the file, as well as the directory distribution, would work.  Not only would the integrity of the files be a side benefit to the naming scheme (easily checked), but you'll have a reasonably even distribution throughout the directory hierarchy.  So if you have a file named "f6a5b1236dbba1647257cc4646308326.jpg" you'd store it in "/f/6" (or as deep as you require).  2 levels deep gives 256 directories, or just under 4000 files per directory for the initial 1m files.  It would also be very easy to automate the redistribution to a deeper scheme.
Comment: very interesting! splitting the filename ... i didn't thought of that. i assume this is the elegant way of doing it :-?
Comment: I definitely agree on using the filessystem and creating an artficial identifier to "slice" up into folder names. But you should also try to get a random distribution of identifiers, i.e. do not use a sequence number. That would allow you have a more balanced tree of folders. In addition, with random distribution you can more easily partition the tree across multiple filesystems. I'd also use a ZFS based SAN with dedup turned on and a sparse volume for each filesystem. You could still use NTFS by using iSCSI to access the SAN.
Comment: +1 I just noticed this answer was similar to the one I just posted.
Comment: I also use this technique, with two optional variation 1. I convert the integer id in base-36 to have a more compact alphanumeric string, you can split digits by two (1296 possibilities)  2. I revert the string to have the less significant (right-most) digit on the top of the folder tree, in this way files fall evenly between folders
Comment: If you go from right to left in step 2 the files are evenly distributed. Also you don't have to worry that you are not filling up with enough zeroes as you can an unlimited number of files
Comment: I use md5sum 3-directory structure and it works great.  As mentioned, it is a decent distribution and the added benefit of dedupe for exact duplicates rather than linking inodes of dupes every day as I used to do.
Comment: It truly depends on you software architecture, some applications are better off with an equal distribution of files, while others are better with a sequence number. it really depends on your goal that you're trying to achieve.
Comment: this is a great answer and works well for a while, but if you're crazy about performance, it's important to group those "small files" into multiple blocks of bigger files, so you can have less disk access for things like file length, etc.. File.length can take 100ms on heavy used systems.
Comment: @Jha, The zero prefix is an optional step right?
Comment: @RafaelSanches I don't see how concatenating files into one big files saves you on disk access? Can you refer me to some article that goes in depth, perhaps one you wrote?
Comment: An issue with this approach is that file names are predictable.
If the files are accesible from the internet, a malicious hacker could enumerate them all with ease.
Answer: I'm going to put my 2 cents worth in on a piece of negative advice: Don't go with a database.
I've been working with image storing databases for years: large (1 meg->1 gig) files, often changed, multiple versions of the file, accessed reasonably often. The database issues you run into with large files being stored are extremely tedious to deal with, writing and transaction issues are knotty and you run into locking problems that can cause major train
wrecks. I have more practice in writing dbcc scripts, and restoring tables from backups than any normal person should ever have.
Most of the newer systems I've worked with have pushed the file storage to the file system, and relied on databases for nothing more than indexing. File systems are designed to take that sort of abuse, they're much easier to expand, and you seldom lose the whole file system if one entry gets corrupted.
Comment: Have you looked at SQL 2008's FILESTREAM data type?  It's a cross between database and file system storage.
Comment: +1 ... a filesystem is kind of a "database" anyway (ntfs for sure), so why make it overly complicated.
Comment: yes. note taken !
Comment: +1 on sticking with file server rather than a database as you are doing fast and infrequent IO operations.
Comment: What if you're just storing a few hundred docs or pics per database - any downside to using database for storage?
Comment: Not all filesystems are equal. Some are robust and identical to databases, some will give you corrupt data upon power loss.
Answer: I think most sites that have to deal with this use a hash of some sort to make sure that the files get evenly distributed in the folders.
So say you have a hash of a file that is something like this 515d7eab9c29349e0cde90381ee8f810
You could have this stored in the following location and you can use how ever many levels deep you need to keep the number of files in each folder low.
\51\5d\7e\ab\9c\29\349e0cde90381ee8f810.jpg
I've seen this approach taken many times.  You still need a database to map these file hashes to a human readable name and what ever other metadata you need to store.  But this approach scales pretty well b/c you can start to distribute the hash address space between multiple computers and or storage pools, etc.
Comment: Git uses a similar approach: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects (to back this answer up)
Answer: Ideally, you should run some tests on random access times for various structures, as your specific hard drive setup, caching, available memory, etc. can change these results.
Assuming you have control over the filenames, I would partition them at the level of 1000s per directory.  The more directory levels you add, the more inodes you burn, so there's a push-pull here.
E.g.,
/root/[0-99]/[0-99]/filename
Note, http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx has more details on NTFS setup.  In particular, "If you use large numbers of files in an NTFS folder (300,000 or more), disable short-file name generation for better performance, and especially if the first six characters of the long file names are similar."
You should also look into disabling filesystem features you don't need (e.g., last access time).  http://www.pctools.com/guides/registry/detail/50/
Comment: +1 for disabling 8.3 filename generation and last access time; those were the first thing that came to mind when I read "huge number of [files]" and "NTFS" (Windows).
Comment: link down........................
Answer: Whatever you do, don't store them all in one directory.
Depending on the distribution of the names of these images, you could create a directory structure where you have single letter top level folders where you would have another set of subfolders for the 2nd letter of images etc.
So:
Folder img\a\b\c\d\e\f\g\ would contain the images starting with 'abcdefg' and so on.
You could introduce your own appropriate depth required.
The great thing about this solution is that the directory structure effectively acts like a hashtable/dictionary. Given an image file name, you will know its directory and given a directory, you will know a subset of images that go there.
Comment: Doing a\b\c\d\e\f\g makes the directory structure very deep and every directory contains only few files. Better to use more that one letter per directory level e.g. ab\cd\ef\ or abc\def\ . Directories also take up space from disk so you do not want too many of them.
Comment: And +1 for "don't store them all in one directory". I'm supporting a legacy system that has put over 47000 files on a server in a single folder, and it takes about a minute for Explorer just to open the folder.
Comment: I had to support an application that had 4+million files all in one directory; it worked surprisingly well, but you could NEVER get explorer to open the folder, it would continually be sorting the new additions. +1 for NTFS being able to handle it without dying.
Comment: That's a generally accepted solution of how to physically store them. Clearly generating the image URL's is something that can be easily done dynamically based on the image file name. Also, to serve them up, you could even introduce img-a, img-b subdomains on the images server if you wanted to, to speed up loading times.
Comment: \a\b\c\d\e\f\ i am doing now, i was thinking there is a wise way of doing this.
Comment: Wim - that's exactly what i am doing now, just thought there are some other folks who've hit this problem.
Comment: You might get better distribution by using the last character (or two, or three) rather than the first.
Comment: @Mark The point is illustrative. It depends on the distribution, as I mentioned.
Comment: Yep. Seen it too. :-o
Comment: It's an illustration - the concept remains the same. It doesn't necessarily make your directory structure deep as it also depends on filename length. Just because I started with a,b,c doesn't mean we need 26 levels.
Comment: The problem with this approach is that you get hot spots.  For instance almost no files will start with the letter z, u, q, etc.  Better to use a file hash algorithm to evenly spread the files between your folders.
Comment: @SqlACID, 4 million in an NTFS folder? What's the max then?
Answer: We have an photo store system with 4 million images. We use database only for meta data and all images are stored on the file system using an inversed naming system, where folder names are generated from last digit of the file, last-1, and so on. e.g.:
000001234.jpg is stored in directory structure like 4\3\2\1\000001234.jpg.
This scheme works very well with identity index in the database, because it fills evenly whole directory structure.
Answer: I would store these on the file system but it depends on how fast will the number of files grow. Are these files hosted on the web? How many users would access these file? These are the questions that need to be answered before I could give you a better recommendation. I would also look at Haystack from Facebook, they have a very good solution for storing and serving up images.
Also if you choose file system you will need to partition these files with directories. I been looking at this issue and proposed a solution but its not a perfect one by any means. I am partitioning by hash table and users you can read more on my blog.
Comment: the images are not meant for frequent access. so there is no problem with this.
their number will grow quite fast. i assume there will be the 1mil. mark in 1 month.
Comment: i'm interested in the programmer view so that i don't overthink this too much
Comment: So if you do not need fast access Haystack is probably not for you. Using Directories for Partitions is the simplest solution in my view.
Answer: The new MS SQL 2008 has a new feature to handle such cases, it's called the FILESTREAM. Take a look:
Microsoft TechNet FILESTREAM Overview
Answer: Quick point, you don't need to store a file path in you DB. You can just store a numeric value, if your files are named in the way you describe. Then using one of the well-defined storage schemes already discussed, you can get the index as a number and very quickly find the file by traversing the directory structure.
Comment: :-? good quick point. just that now i don't have an algorithm for generating the path.
Answer: Will your images need to be named uniquely?  Can the process that generates these images produce the same filename more than once? Hard to say without knowing what device is creating the filename but say that device is 'reset' and upon restart it begins naming the images as it did the last time it was 'reset' - if that is such a concern..
Also, you say that you will hit 1 million images in one month's time.  How about after that?  How fast will these images continue to fill the file system?  Will they top off at some point and level out at about 1 million TOTAL images or will it continue to grow and grow, month after month?
I ask because you could begin designing your file system by month, then by image.  I might be inclined to suggest that you store the images in such a directory structure:
imgs\yyyy\mm\filename.ext

where: yyyy = 4 digit year
         mm = 2 digit month

example:  D:\imgs\2009\12\aaa0001.jpg
          D:\imgs\2009\12\aaa0002.jpg
          D:\imgs\2009\12\aaa0003.jpg
          D:\imgs\2009\12\aaa0004.jpg
                   |
          D:\imgs\2009\12\zzz9982.jpg
          D:\imgs\2010\01\aaa0001.jpg (this is why I ask about uniqueness)
          D:\imgs\2010\01\aab0001.jpg

Month, year, even day is good for security type images.  Not sure if this is what you are doing but I did that with a home security camera that snapped a photo every 10 seconds... This way your application can drill down to specific time or even a range where you might think the image was generated.  Or, instead of year, month - is there some other "meaning" that can be derived from the image file itself?   Some other descriptors, other than the date example I gave?
I would not store the binary data in the DB.  Never had good performance / luck with that sort of thing.  Cant imagine it working well with 1 million images.  I would store the filename and that is it.  If they are all going to be JPG then dont even store the extension.  I would create a control table that stored a pointer to the file's server, drive, path, etc.   This way you can move those images to another box and still locate them.  Have you a need to keyword tag your images?   If so then you would want to build the appropriate tables that allow that sort of tagging.
You / others may have addressed these ideas while I was replying..  Hope this helps..
Comment: 1.all files will be named uniquely
2.the system will grow and grow at first it will get out arround 1mil images and then grow at a rate of a couple tens of thousands per month.
3.there will be some sort of tagging of the files at some point in the future, that's why i want to store some sort of identification data in the db.
Answer: I would be inclined to create a date based folder structure, e.g. \year\month\day, and use timestamps for the filenames. If necessary, the timestamps can have an additional counter component if the images are to be created so fast that there may be more than one within a millisecond. By using a most significant to least significant sequence for the naming sorting, finding and maintenance are a breeze. e.g. hhmmssmm[seq].jpg
Answer: I am involved in a project that stores 8.4 million images in the course of a year for documenting status of various devices. More recent images are accessed more frequently, and older images are rarely sought unless a condition was discovered which prompted someone to dig into the archives.
My solution, based on this usage, was to incrementally zip the images into compressed files. The images are JPGs, each approximately 20kB and do not compress much, so the ZIP compression scheme is none. This is done merely for concatenating them into one filesystem entry which greatly helps NTFS in terms of speed when it comes to moving them from drive to drive, or looking through the list of files.
Images older than a day are combined into a "daily" zip; zips older than a month are combined into their respective "monthly" zip; and finally anything over a year is no longer needed and consequently deleted.
This system works well because users can browse the files (either via the operating system or a number of client applications) and everything is named based on device names and timestamps. Generally a user knows these two pieces of information and can quickly locate any one of the millions of images.
I understand this is probably not related to your particular details, but I thought I would share.
Answer: While I haven't served pictures on that scale, I've previously written a small gallery app for serving ~25k pictures on a 400MHz machine w. 512 MB of RAM or so. Some experiences;

Avoid a relational databases at all costs; while databases, no doubt, are smart about handling data, they aren't designed for such use (we got specialized, hierarchal key-value databases for that called file systems). While I have nothing more than a hunch, I'd wager that the DB cache goes out the window, if you throw really large blobs at it. While my available hardware was in the small end, not touching the DB at all on image lookup gave orders of magnitude better speed.
Research how the file system behaves; on ext3 (or was it ext2 at the time - can't remember), the limit of being able to efficiently look up sub-directories and files was around the 256  mark; so having only that many files and folders in any given folder. Again, noticeable speedup. While I do not known about NTFS, stuff like XFS (which uses B-trees, as far as I remember) is extremely fast, simply because they can do lookups extremely fast.
Distribute data evenly; when I experimented with the above, I tried to distribute the data evenly over all directories (I did an MD5 of the URL and used that for directories; /1a/2b/1a2b...f.jpg). That way it takes longer to hit whatever performance limit there is (and file system cache is void at such large datasets anyway). (contrarily, you might want to see where the limits are early on; then you want to throw everything in the first available directory.
Answer: Perhaps a creation date based naming scheme - either including all the info in the file name or (better for browsing later) splitting it up in directories. I can think of the following, depending on how often you generate images:

Several images generated each day: Year/Month/Day/Hour_Minute_Second.png
A couple a month: Year/Month/Day_Hour_Minute_Second.png

etc. You get my point... =)
Comment: they are not continuously generated over time, so some folders will become fat and others stay... slim :))
Comment: Well, you obviously don't have to create *each* folder, just because you're following this scheme. You could even have `Year/Month/Day/Hour/Minute` - decide how many levels of folders you need depending on how often the images are generated *when the rate is the highest* - and then just don't create folders that would be left empty.
Answer: Are you considering disaster recovery?
Some of the proposed solutions here end up mangling the file name (such that if the physical file was moved you would lose track of what file it really is). I recommend maintaining a unique physical file name so that if your master list of file locations gets corrupted, you can regenerate it with a small shell, er, powershell, script ;)
From what I read here it sounds like all these files would be stored on one file system. Consider storing them across multiple file systems on multiple machines. If you have the resources, determine a system of storing each file on two different machines in case you lose a power supply and the replacement is 2 days out.
Consider what kinds of procedures you would need to create to migrate files between machines or file systems. The ability to do this with your system is live and online may save you considerable headache down the road.
You might consider using a GUID as a physical file name instead of an incremental number in case your incremental number counter (the database identity column?) gets messed up.
If appropriate, consider using a CDN such as Amazon S3.
Answer: Might be late to the game on this. But one solution (if it fits your use-case) could be file name hashing. It is a way to create an easily reproducible file path using the name of the file while also creating a well distributed directory structure. For example, you can use the bytes of the filename's hashcode as it's path:
String fileName = "cat.gif";
int hash = fileName.hashCode();
int mask = 255;
int firstDir = hash & mask;
int secondDir = (hash >> 8) & mask;

This would result in the path being:
/172/029/cat.gif

You can then find cat.gif in the directory structure by reproducing the algorithm.
Using HEX as the directory names would be as easy as converting the int values:
String path = new StringBuilder(File.separator)
        .append(String.format("%02x", firstDir))
        .append(File.separator)
        .append(String.format("%02x", secondDir)
        .toString();

Resulting in:
/AC/1D/cat.gif

I wrote an article about this a few years ago and recently moved it to Medium. It has a few more details and some sample code: File Name Hashing: Creating a Hashed Directory Structure. Hope this helps!
Comment: We store 1.8 billion items using something similar. It works well. Use a hash that's fast and has low collisions rates and you're set.
Answer: If you are on windows how about on an exFat filessytem
http://msdn.microsoft.com/en-us/library/aa914353.aspx
it was designed with storing media files in mind, and available now.
Answer: If they ALL are not immediately required and you can generate them on-the-fly and these are small images, why not implement an LRU memory- or disk-cache above your image generator?
This could save you from the storage and keep the hot images to be served from mem?
Answer: I just ran a test on zfs because I love zfs, and I had a 500gig partition that I had compression on. I wrote a script that generated 50-100k files and placed them in nested directories 1/2/3/4/5/6/7/8  (5-8 levels deep) and let it run for I think 1 week. (it wasn't a great script.)
It filled up the disk and ended up having about 25 million files or so. Access to any one file with a known path was instant. Listing any directory with a known path was instant.
Getting a count of the list of files however (via find) took 68 hours.
I also ran a test putting a lot of files in one directory. I got up to about 3.7 million files in one directory before I stopped. Listing the directory to get a count took about 5 minutes. Deleting all the files in that directory took 20 hours. But lookup and access to any file was instant.
Answer: I see other mention a database, but see no mention of that in your post. In any case, my opinion on this particular point is: either stick to a database or file system. If you have to mix the two, be careful about it. Things get more complicated. But you might have to. Storing a million photos in a database does not sound the best idea.
You might be interested by the following specification, most digital cameras follow it to manage file storage:
https://en.wikipedia.org/wiki/Camera_Image_File_Format
Essentially, a folder is created, such as 000OLYMPUS and photos are added to that folder (for example DSC0000.RAW). When the file name counter reaches DSC9999.RAW a new folder is created (001OLYMPUS) and image are added again, resetting the counter, possibly with a different prefix (ex: P_0000.RAW).
Alternatively you could also create folders based on parts of the file name (already mentioned several times). For example, if you photo is named IMG_A83743.JPG, store it at IMG_\A8\3\IMG_A83743.JPG. It is more complicated to implement but will make your files easier to find.
Depending on the filesystem (this will require some research), you might be able to just dump all the images in a single folder, but, in my experience, this would usually cause performance issues.
Answer: You might want to be look at ZFS (file system, volume manager from Sun)
Regards
Answer: A clean way to generate the path from a large number is to easily convert it into hex then split it!
for example 1099496034834 > 0xFFFF1212 > FF/FF/12/12
public string GeneratePath(long val)
{
    string hex = val.ToString("X");
    hex=hex.PadLeft(10, '0');
    string path="";
    for(int i=0; i<hex.Length; i+=2 )
    {
        path += hex.Substring(i,2);
        if(i+2<hex.Length)
            path+="/";
    }
    return path;
}

Store and load :
public long Store(Stream doc)
{
   var newId = getNewId();
   var fullpath = GeneratePath(newId)
   // store into fullpath
   return newId;
}

public Stream Load(long id)
{
   var fullpath = GeneratePath(newId)
   var stream = ...
   return stream;
}

Full source codes : https://github.com/acrobit/AcroFS
Answer: Unfortunately filesystems are very bad (performance with many files per directory or deep directory trees, checking times on restart, reliability) at managing lots of small files, so the solution above that involves ZIP files is best if you want to use a filesystem.
Using a database manager is by far the best option; a simple one like BDB or GDBM for example; even a relatrional DBMS like MySQL would be better. Only lazy people who don't understand filesystems and databases (e.g. those who dismiss transactions) tend to use filesystems as databases (or somewhat more rarely, viceversa).
Answer: How about a database with a table containing an ID and a BLOB to store the image? Then you can add new table(s) whenever you want to associate more data elements with a photo.
If you're expecting to scale, why not scale now? You'll save time both now and later IMO. Implement the database layer once, which is fairly easy to start with. Or implement some thing with folders and filenames and blah blah blah, and later switch to something else when you start blowing up MAX_PATH.
Comment: Been there, done that, have the scars to prove it. Databases that store images in large numbers are cranky almost beyond belief, and require inordinate amounts of maintenance. Much better to store them in the file system unless you have a specific need that can only be answered by a database (ours was version tracking.)
Comment: Oh God No. Please dont use a database as large BLOB storage.
Comment: And there are lots of utilities to deal with files and file systems, few to none to deal with files within a database.
Comment: Eek. Didn't know that databases (still?) have so many problems with BLOBs.
Comment: How can such a bad solution that has so many comments still have a +1? no offence to the OP (I see it came from SO) but the downvote button is here for a reason!
Comment: Because it's not a universally bad solution. Sure,  for most  people I wouldn't recommend it,  but there is a small subset of applications for which the benefits (yes there are some benefits)  are actually quite important, like acid compliance, or replication,  or as someone else mentioned having versioning.  Though admittedly,  this particular answer really didn't sell it very well and seemed to be recommending it for a the wrong reasons.


Chrome Extension Overlay Page

Question: I create a Chrome Extension and would like to know:
How can I create an overlay over an existing webpage in a chrome extension?
I don't want to use a popup, I would rather prefer a banner at the top or bottom of the page.
I did read the Google Chrome Extension documentation but couldn't find the solution for my problem in it. Can you give me a hint?
Comment: There's no such API, so you'll have to add a standard DOM element to the page and adjust the CSS.
Answer: You can create a content script that injects an  on the top/bottom of the web page. That is on all http, and https websites.
Note: This code does not work on the local chrome:// and Chrome Web Store pages.
For example in your content.js:
var newiframe = document.createElement("iframe");
newiframe.setAttribute("src", chrome.runtime.getURL("html/panel.html"));
newiframe.style.width = "100%";
newiframe.style.top = "0";
newiframe.style.left = "0";
newiframe.style.right= "0";
newiframe.style.position = "absolute";
newiframe.style.zIndex = "500";
newiframe.style.height = "80px";
document.body.appendChild(newiframe);

And in your manifest.json:
{
  "manifest_version": 3,
  "name": "__MSG_name__",
  "short_name": "Proper Menubar",
  "version": "1.0",
  "description": "__MSG_description__",
  "content_scripts": [
    {
      "matches": ["*://*/*"],
      "js": ["js/content.js"],
      "run_at": "document_end"
    }
  ],
  "web_accessible_resources": [{
    "resources": [ "html/panel.html",
    "js/script.js",
    "css/body.css"],
    "matches": ["<all_urls>"],
    "use_dynamic_url": true
  }],
  "permissions": [ "activeTab" ]
}

panel.html:

Hello there

Comment: Thank you so much for your very detailled answer. That helped me a lot! I was actually quite close...But sometimes close is just not enough ;)
It works fine for me!
Comment: Great to hear it.

Chrome Extension Overlay Page


Does making a jump discontinuity undefined actually make the function continuous?

Question: I'll do this with an example, although I don't know how to do the big fancy piecewise bracket curly brace thing:
Consider function $f$ where
$$f(x)=\begin{cases} -1 & \text{if }x \leq 0,\\
1&\text{if }x > 0.
\end{cases}$$
And a slight modification of the same function:
$$g(x)=\begin{cases} -1 & \text{if }x < 0,\\
1&\text{if }x > 0.
\end{cases}$$The only difference between $f$ and $g$ is that $0$ is excluded from the domain of $g$. Here is a picture to show both graphs in case that is easier:

My question:
For $f(x)$, since $f(0)$ is defined, we can say $0$ is part of $f$'s domain. However, the two-sided limit does not exist at $x=0$, which means we can say $f$ is discontinuous at $x=0$.
For $g(x)$, we have $x=0$ excluded from the domain. Does this, somewhat counter-intuitively, make $g$ a continuous function? Since the domain is split into something that looks like $(-\infty, 0) \cup (0, \infty)$, each of those two intervals are continuous on their own, and a function is continuous iff it is continuous at all points in its domain.
Is that right? $f$ is discontinuous, but $g$ is continuous?
Comment: `somewhat counter-intuitively` *Both* $\,f\,$ and $\,g\,$ are continuous, in fact identical, on $\,\mathbb{R} \setminus \{0\}\,$. Then you extend the domain to include one extra point, in this case $\,0\,$. Not sure why you'd find that it would be `intuitively` expected that continuity were preserved on the newly extended domain.
Answer: Yes, $f$ is discontinuous but $g$ is continuous. But the domain of $g$ doesn't just "look like" $(-\infty,0)\cup(0,+\infty)$ (you wrote $\cap$ here, but I suppose that that's a typo); it is that set. By removing from the domain of $f$ the only point at which it was discontinuous, you turned it into a continuous function.
Comment: Fixed my typo, thanks for the catch


Menu items with Userpoints total

Question: Is it possible to show the current total userpoints a user have next to menu item?
example menu: Home | Downloads | My Points (24) | Logout
if it's possible, how would one do this (beginner)
Answer: This snippet will return you user points :
<code><?php
print userpoints_get_current_points($user->uid);
?>
</code>
But to Place this PHP snippet in menu link, you may need Menu views and PHP Menu Modules
Comment: Thank you, it turns out you only need the Menu Views modules only to show the total amount of Userpoints as it already integrated inside views. Can't seem to upvote your anwers right now, I dont seem to have enough reputation. Will upvote it later :)
Comment: Glad that helped you :)

Menu items with Userpoints total


Truffle error: Error encountered, bailing. Network state unknown. Review successful transactions manually

Question: Every time I try to deploy truffle migrate --reset my Organization.sol contract, I get the following error:

Error encountered, bailing. Network state unknown. Review successful transactions manually.

I Googled the error and apparently this is due to contract size. My Organization.sol is not even that big; I've seen bigger contracts before.
Here is my contract: https://github.com/ulixir/eth-task-contracts/blob/master/contracts/Organization.sol
Any clues?
Comment: it works fine with [StandardToken.sol](https://github.com/OpenZeppelin/zeppelin-solidity/blob/master/contracts/token/StandardToken.sol) and truffle version v4.0.0-beta.2.
Comment: One hint: When you set gas: 4700000 (limit on ropsten) in truffle.js for development. Can you reproduce it for local ganache-cli deployment? If that's the case it's definitely contract size. Your github link seems to be down so meanwhile...
Answer: Do you also see an error this error

Error: insufficient funds for gas * price + value

In that case, the ethereum account you are trying to use may not have enough ether to perform the migration.
Ether can be added to an account by calling miner.start() from the geth console or by allocating ether to the account using the genesis.json file. You can read more about it here.
If you decide to use the genesis file, take the following steps after allocating data to your ethereum account:

remove the old geth db using geth --datadir "<your datadir>" removedb.
Reinitialize your geth node using the modified genesis file  geth <your other params here> init <yourGenesisFile>
within the geth console of the new node, make sure you start the miner miner.start()

When done, try the migration again, it should work now.
If you decided to mine ether without modifying the genesis file, start the mining and give your ethereum account some time to mine enough ether. How long this takes depends on the difficulty level you set on your genesis file.
Answer: When you are using this command truffle migrate --reset it is trying to deploy Migration.sol contract also with your Organization.sol .You can see the migration script in migration folder. The error

Error encountered, bailing. Network state unknown. Review successful transactions manually.

means that it cannot determine which contracts are deployed.And telling you to check manually.To check which contracts are deployed you can use this command truffle network.


Is PCA optimization convex?

Question: The objective function of Principal Component Analysis (PCA) is minimizing the reconstruction error in L2 norm (see section 2.12 here. Another view is trying to maximize the variance on projection. We also have an excellent post here: What is the objective function of PCA?).
My question is that is PCA optimization convex? (I found some discussions here, but wish someone could provide a nice proof here on CV).
Comment: I think you need to be specific about what you mean by "PCA optimization."  One standard formulation is to maximize $x^\prime\mathbb{A}x$ subject to $x^\prime x=1$. The problem is that convexity doesn't even make sense: the domain $x^\prime x=1$ is a sphere, not a Euclidean space.
Comment: I would refer you to any definition of "convex" that you are familiar with.  Don't they all involve a concept of points in the domain of a function lying "between" other points? That's worth remembering, because it reminds you to consider the geometry of the domain of a function as well as any algebraic or analytic properties of the function values. In that light, it occurs to me that the variance-maximizing formulation can be slightly modified to make the domain convex: simply require $x^\prime x\le1$ rather than $x^\prime x=1$. The solution is the same--and the answer becomes quite clear.
Comment: No. You are *maximizing* a convex function (under constraints).
Comment: @whuber thanks for your comment, I may not able to clarify the question due to limited knowledge. I may wait for some answers can help me to clarify the question at same time.
Answer: No, the usual formulations of PCA are not convex problems.  But they can be transformed into a convex optimization problem.
The insight and the fun of this is following and visualizing the sequence of transformations rather than just getting the answer: it lies in the journey, not the destination.  The chief steps in this journey are

Obtain a simple expression for the objective function.

Enlarge its domain, which is not convex, into one which is.

Modify the objective, which is not convex, into one which is, in a way that obviously does not change the points at which it attains its optimal values.

If you keep close watch, you can see the SVD and Lagrange multipliers lurking--but they're just a sideshow, there for scenic interest, and I won't comment on them further.

The standard variance-maximizing formulation of PCA (or at least its key step) is
$$\text{Maximize }f(x)=\ x^\prime \mathbb{A} x\ \text{ subject to }\ x^\prime x=1\tag{*}$$
where the $n\times n$ matrix $\mathbb A$ is a symmetric, positive-semidefinite matrix constructed from the data (usually its sum of squares and products matrix, its covariance matrix, or its correlation matrix).
(Equivalently, we may try to maximize the unconstrained objective $x^\prime \mathbb{A} x / x^\prime x$.  Not only is this a nastier expression--it's no longer a quadratic function--but graphing special cases will quickly show it is not a convex function, either.  Usually one observes this function is invariant under rescalings $x\to \lambda x$ and then reduces it to the constrained formulation $(*)$.)
Any optimization problem can be abstractly formulated as

Find at least one $x\in\mathcal{X}$ that makes the function $f:\mathcal{X}\to\mathbb{R}$ as large as possible.

Recall that an optimization problem is convex when it enjoys two separate properties:

The domain $\mathcal{X}\subset\mathbb{R}^n$ is convex.  This can be formulated in many ways.  One is that whenever $x\in\mathcal{X}$ and $y\in\mathcal{X}$ and $0 \le \lambda \le 1$, $\lambda x + (1-\lambda)y\in\mathcal{X}$ also.  Geometrically: whenever two endpoints of a line segment lie in $\mathcal X$, the entire segment lies in $\mathcal X$.

The function $f$ is convex.  This also can be formulated in many ways.  One is that whenever $x\in\mathcal{X}$ and $y\in\mathcal{X}$ and $0 \le \lambda \le 1$, $$f(\lambda x + (1-\lambda)y) \ge \lambda f(x) + (1-\lambda) f(y).$$ (We needed $\mathcal X$ to be convex in order for this condition to make any sense.) Geometrically: whenever $\bar{xy}$ is any line segment in $\mathcal X$, the graph of $f$ (as restricted to this segment) lies above or on the segment connecting $(x,f(x))$ and $(y,f(y))$ in $\mathbb{R}^{n+1}$.
The archetype of a convex function is locally everywhere parabolic with non-positive leading coefficient: on any line segment it can be expressed in the form $y\to a y^2 + b y + c$ with $a \le 0.$

A difficulty with $(*)$ is that $\mathcal X$ is the unit sphere $S^{n-1}\subset\mathbb{R}^n$, which is decidedly not convex.  However, we can modify this problem by including smaller vectors.  That is because when we scale $x$ by a factor $\lambda$, $f$ is multiplied by $\lambda^2$. When $0 \lt x^\prime x \lt 1$, we can scale $x$ up to unit length by multiplying it by $\lambda=1/\sqrt{x^\prime x} \gt 1$, thereby increasing $f$ but staying within the unit ball $D^n = \{x\in\mathbb{R}^n\mid x^\prime x \le 1\}$.  Let us therefore reformulate $(*)$ as
$$\text{Maximize }f(x)=\ x^\prime \mathbb{A} x\ \text{ subject to }\ x^\prime x\le1\tag{**}$$
Its domain is $\mathcal{X}=D^n$ which clearly is convex, so we're halfway there.  It remains to consider the convexity of the graph of $f$.
A good way to think about the problem $(**)$--even if you don't intend to carry out the corresponding calculations--is in terms of the Spectral Theorem.  It says that by means of an orthogonal transformation $\mathbb P$, you can find at least one basis of $\mathbb{R}^n$ in which $\mathbb A$ is diagonal: that is,
$$\mathbb {A = P^\prime \Sigma P}$$
where all the off-diagonal entries of $\Sigma$ are zero.  Such a choice of $\mathbb{P}$ can be conceived of as changing nothing at all about $\mathbb A$, but merely changing how you describe it: when you rotate your point of view, the axes of the level hypersurfaces of the function $x\to x^\prime \mathbb{A} x$ (which were always ellipsoids) align with the coordinate axes.
Since $\mathbb A$ is positive-semidefinite, all the diagonal entries of $\Sigma$ must be non-negative.  We may further permute the axes (which is just another orthogonal transformation, and therefore can be absorbed into $\mathbb P$) to assure that $$\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_n \ge 0.$$
If we let $x=\mathbb{P}^\prime y$ be the new coordinates $x$ (entailing $y=\mathbb{P}x$), the function $f$ is
$$f(y) = y^\prime \mathbb{A} y = x^\prime \mathbb{P^\prime A P} x = x^\prime \Sigma x = \sigma_1 x_1^2 + \sigma_2 x_2^2 + \cdots + \sigma_n x_n^2.$$
This function is decidedly not convex!  Its graph looks like part of a hyperparaboloid: at every point in the interior of $\mathcal X$, the fact that all the $\sigma_i$ are nonnegative makes it curl upward rather than downward.
However, we can turn $(**)$ into a convex problem with one very useful technique.  Knowing that the maximum will occur where $x^\prime x = 1$, let's subtract the constant $\sigma_1$ from $f$, at least for points on the boundary of $\mathcal{X}$.  That will not change the locations of any points on the boundary at which $f$ is optimized, because it lowers all the values of $f$ on the boundary by the same value $\sigma_1$.  This suggests examining the function
$$g(y) = f(y) - \sigma_1 y^\prime y.$$
This indeed subtracts the constant $\sigma_1$ from $f$ at boundary points, and subtracts smaller values at interior points.  This will assure that $g$, compared to $f$, has no new global maxima on the interior of $\mathcal X$.

Let's examine what has happened with this sleight-of-hand of replacing $-\sigma_1$ by $-\sigma_1 y^\prime y$.  Because $\mathbb P$ is orthogonal, $y^\prime y = x^\prime x$.  (That's practically the definition of an orthogonal transformation.)  Therefore, in terms of the $x$ coordinates, $g$ can be written
$$g(y) = \sigma_1 x_1 ^2 + \cdots + \sigma_n x_n^2 - \sigma_1(x_1^2 + \cdots + x_n^2) = (\sigma_2-\sigma_1)x_2^2 + \cdots + (\sigma_n - \sigma_1)x_n^2.$$
Because $\sigma_1 \ge \sigma_i$ for all $i$, each of the coefficients is zero or negative.  Consequently, (a) $g$ is convex and (b) $g$ is optimized when $x_2=x_3=\cdots=x_n=0$.  ($x^\prime x=1$ then implies $x_1=\pm 1$ and the optimum is attained when $y = \mathbb{P} (\pm 1,0,\ldots, 0)^\prime$, which is--up to sign--the first column of $\mathbb P$.)

Let's recapitulate the logic.  Because $g$ is optimized on the boundary $\partial D^n=S^{n-1}$ where $y^\prime y = 1$, because $f$ differs from $g$ merely by the constant $\sigma_1$ on that boundary, and because the values of $g$ are even closer to the values of $f$ on the interior of $D^n$,  the maxima  of $f$ must coincide with the maxima of $g$.
Comment: +1 Very nice. I edited to fix one formula to what I think you intended (but please check). Apart from that, I found the sentence "That won't change any boundary values at which f is optimized" to be confusing at first, because the boundary values *do* change: you are subtracting $\sigma_1$. Maybe it makes sense to reformulate a bit?
Comment: (+1) In your answer, you seem to define a convex function to be what most people would consider to be a concave function (perhaps since a convex optimization problem has a convex domain and a *concave* function over which a *maximum* is computed (or a *convex* function over which a *minimum* is computed))
Comment: @amoeba It's a subtle argument.  Note, however, that the *new* maxima--those of $g$--are found to occur only on the boundary.  That rules out your counterexamples.  Another point worth noting is that in the end we don't really care whether new local (or even global) maxima happen to show up in the interior of $\mathcal X$, because we are originally concerned only about local maxima on its boundary.  We are therefore free to alter $f$ in any way that will not make any of those local boundary maxima move or disappear.
Comment: Yes, I agree. It does not matter how $f$ is modified on the inside, if the resulting $g$ is "convex" and happens to have maxima on the boundary. Your $g$ does happen to have maxima on the boundary, and this makes the whole argument work. Makes sense.
Comment: @amoeba Right on all counts; thank you.  I have amplified the discussion of that point.
Comment: @Ben That's right.  I probably should adhere more closely to convention (which, as you suggest, is to focus on *minimization*, which incurs no loss of generality).   Since I have plans to add some links, I'll think about changing the terminology a little.
Comment: On re-reading, I don't quite understand the following crucial step: "[This] subtracts smaller values at interior points. This will assure that g, compared to f, has no new global maxima on the interior of X." The original function $g$ has maximum on X and smaller values inside. To get $f$, you subtract a constant on X and subtract something smaller inside. I don't think that this, by itself, is enough to claim that $f$ cannot get a new global maximum somewhere inside. Consider $g=x^2$ on $[-1,1]$, subtract $2$ on the boundary (obtaining 1) and $0<2$ inside; new maximum is at $x=0$.
Comment: Nice answer! Thx. BTW, could you provide the references of your answer? Is there any paper/ lecture note taking about this problem?
Comment: @stander I'm sorry I have no references.  I created this argument in response to the question, which I have not seen addressed elsewhere.
Comment: This question is a bit old but I came into it now. After reading your answer I still fail to see it. The first step is clear to me. Simply changing the constraint to go from the sphere (non convex) to the ball (convex). But I am having some problems with the second step. As far as I understand it, in order for g(y) to achieve the same solution as f(y), you require to constraint it to the sphere, otherwise, if you work on the ball, g(y) would be maximized when (x1,...xn)=0. I would be grateful if you could indicate what am I not understanding.
Comment: @ÁlvaroMéndezCivieta You might want to read this comment thread, because amoeba had the same question, which was resolved.  If that doesn't do the trick, consider graphing the objective functions in one dimension.
Comment: Thank you for your answer. I read the thread but still I do not understand. You modify the domain of f() so it is now convex, and then define the function g() which is convex. But as you say in your answer, g() needs to be optimized in the boundary, rendering again a non convex problem, eg, a convex function (g) in a non convex domain (the boundary). I fail to see how this can be translated into a convex optimization problem.
Comment: @Alvaro I do not modify the domain of $f$ at all, which makes it difficult to respond to your objections.
Comment: I am sorry I was not clear expressing myself. What I meant is that the domain in which the optimization problem is to be solved changed from x^t x = 1 to x^t x <=1. Being the fisrt a radius 1 sphere and the latter a convex domain. But still, when you define the function g() you state that g() is solved in the sphere, which is non-convex. So I fail to understand how this redefines PCA as a convex problem when g() is convex but it is still solved in a non-convex domain.
Comment: @ÁlvaroMéndezCivieta $g$ is defined on the entire ball, not just its surface.
Comment: But if g() is defined on the entire ball and not just the surface, the optimal value of g(y) would be achieved when y=(0,...,0)
Comment: @ÁlvaroMéndezCivieta By construction, $g(0)=f(0)=0.$ Are you perhaps confusing maxima and minima?
Comment: @ÁlvaroMéndezCivieta I have added two figures to illustrate the construction.
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/118680/discussion-between-alvaro-mendez-civieta-and-whuber).
Answer: No.
Rank $k$ PCA of matrix $M$ can be formulated as
$\hat{X} = \underset{rank(X) \leq k}{argmin} \| M - X\|_F^2$
($\|\cdot\|_F$ is Frobenius norm). For derivation see Eckart-Young theorem.
Though the norm is convex, the set over which it is optimized is nonconvex.

A convex relaxation of PCA's problem is called Convex Low Rank Approximation
$\hat{X} = \underset{\|X\|_* \leq c}{argmin} \| M - X\|_F^2$
($\|\cdot\|_*$ is nuclear norm. it's convex relaxation of rank - just like $\|\cdot\|_1$ is convex relaxation of number of nonzero elements for vectors)
You can see Statistical Learning with Sparsity, ch 6 (matrix decompositions) for details.
If you're interested in more general problems and how they relate to convexity, see Generalized Low Rank Models.
Answer: Disclaimer: The previous answers do a pretty good job of explaining how PCA in its original formulation is non-convex but can be converted to a convex optimization problem.
My answer is only meant for those poor souls (such as me) who are not so familiar with the jargon of Unit Spheres and SVDs - which is, btw, good to know.
My source is this lecture notes by Prof. Tibshirani
For an optimization problem to be solved with convex optimization techniques, there are two prerequisites.

The objective function has to be convex.
The constraint functions should also be convex.

Most formulations of PCA involve a constraint on the rank of a matrix.
In these type of PCA formulations, condition 2 is violated. Because, the constraint that $rank(X) = k, $ is not convex. For example, let $J_{11}$, $J_{22}$ be 2 × 2 zero matrices with a single 1 in the upper left corner and lower
right corner respectively. Then, each of these have rank 1, but their average has rank 2.
Comment: Could you please explain what "$X$" refers to and why there is any constraint on its rank?  This doesn't correspond with my understanding of PCA, but perhaps you are thinking of a more specialized version in which only $k$ principal components are sought.
Comment: Yeah, $X$ is the transformed (rotated) data matrix. In this formulation, we seek matrices that are at least of rank $k$. You can refer to the link in my answer for a more accurate description.

Is PCA optimization convex?


Goodness-of-fit test in Logistic regression; which 'fit' do we want to test?

Question: I am referring to the question and its answers: How to compare (probability) predictive ability of models developed from logistic regression? by @Clark Chong and answers/comments by @Frank Harrell. and to the question Degrees of freedom of $\chi^2$ in Hosmer-Lemeshow test and the comments.
I have read the paper D.W. Hosmer, T. Hosmer, S. Le Cessie, S. Lemeshow, "A comparison of goodness-of-fit tests for the Logistic regression model",  Statistics in Medicine, Vol. 16, 965-980 (1997).
After reading I was confused because the question I referred to asks explicitly for "(probability) predictive ability", which is in my opinion not the same as what the goodness-of-fit-tests in the paper supra aim at:
As most of us know, the logistic regression assumes an S-shaped link between the explanatory variables and the probability of success, the functional form for the S-shape is
$P(y=1|_{x_i})=\frac{1}{1+e^{-(\beta_0+\sum_i \beta_i x_i)}}$
Without pretending that there are no shortcomings with the Hosmer–Lemeshow test, I think that we have to distinguish between tests for the (a) '(probability) predictive ability' and (b) 'goodness-of-fit'.
The former's goal is to test whether the probabilities are well predicted, while the goodness-of-fit tests test whether the S-shaped function above is the 'right' function.  More formally:

tests for 'probability predictive ability tests' have a $H_0$ stating that the success probabilities are well predicted by the model;
while for goodness-of-fit tests $H_0$ is (see Hosmer et. al. ) that the S-shaped functional form supra is the correct one.  Hosmer et al. perform simulations where they find the power to detect two types of deviations from the null namely that the link function is wrong or that the exponent in the denominator is not linear.

Obviously, if the above function has the 'right' functional form (so if the tests concludes that we can accept $H_0$ for the goodness-of-fit test), then the predicted probabilities will be fine, ...
First remark
...however, accepting the $H_0$ is a weak conclusion as explained in What follows if we fail to reject the null hypothesis?.
First question
The most important question/remark that I have is that if the goodness-of-fit $H_0$ is rejected, then the conclusion of the test is that the functional form was not the 'right' one, however, does this imply that the probabilities are not well predicted ?
Second question
Furthermore, I want to point to the conclusions of Hosmer et. al; (I cite from the abstract):
''An examination of the performance of the tests when the correct model has a quadratic term but a model containing only the linear term has been fit shows that the Pearson chi-square, the unweighted sum-of-squares,the Hosmer-Lemeshow decile of risk, the smoothed residual sum-of-squares and Stukel's score test,have power exceeding 50 per cent to detect moderate departures from linearity when the sample size is 100 and have power over 90 per cent for these same alternatives for samples of size 500. All tests had no power when the correct model had an interaction between a dichotomous and continuous covariate but only the
continuous covariate model was fit. Power to detect an incorrectly specifed link was poor for samples of size 100. For samples of size 500 Stukel's score test had the best power but it only exceeded 50 per cent to detect an asymmetric link function. The power of the unweighted sum-of-squares test to detect an incorrectly specifed link function was slightly less than Stukel's score test
''
Can I conclude from this which test has more power or that Hosmer–Lemeshow has less power (to detect these specific anomalies) ?
Second remark
The paper by Hosmer et. al. that I referred to supra, compute (simulate) the power to detect specific anomalies (the power can only be computed if an $H_1$ is specified).  This does at my opinion not imply that these results can be generalised to ''all possible alternatives $H_1$'' ?
Comment: Hosmer-Lemeshow is considered obsolete:  https://stats.stackexchange.com/questions/18750/hosmer-lemeshow-vs-aic-for-logistic-regression  or  https://stats.stackexchange.com/questions/180506/hosmer-lemeshow-for-correlated-data  or  https://stats.stackexchange.com/questions/273966/logistic-regression-with-poor-goodness-of-fit-hosmer-lemeshow
Comment: @kjetilbhalvorsen, the link points to an answer that contains an unsupported claim. Where can I see *why* Hosmer-Lemeshow is considered obsolete?
Comment: @Richard Hardy: I added some link which contain explanation!
Comment: @kjetilbhalvorsen, thank you.
Answer: "Goodness of fit" is sometimes used in one sense as the contrary of evident model mis-specification, "lack of fit"; & sometimes in another sense as a model's predictive performance—how well predictions match to observations. The Hosmer–Lemeshow test is for goodness of fit in the first sense, & although evidence of lack of fit suggests predictive performance (GoF in the second sense, measured by say Nagelkerke's $R^2$ or Brier scores) could be improved, you're none the wiser as to how or by how much until you try out specific improvements (typically by including interaction terms, or a spline or polynomial basis for representing continuous predictors to allow for a curvilinear relationship with the logit; sometimes by changing the link).
Goodness-of-fit tests are intended to have reasonable power against a variety of alternatives, rather than high power against a specific alternative; so people comparing of the power of different tests tends to take the pragmatic approach of picking a few alternatives that are thought to be of particular interest to potential users (see for example the frequently cited Stephens (1974), "EDF statistics for goodness of fit & some comparisons", JASA, 69, 347). You can't conclude that one test is more powerful than another against all possible alternatives because it's more powerful against some.
Comment: In general, "goodness of fit" is given too much emphasis IMHO.  A better alternative is to make the model fit up front.  This is done by using regression splines to relax linearity assumptions, and including interactions that would make sense.
Comment: In some cases it can be shown that a test is 'uniformly more powerfull', which means that it is more powerfull for all possible alternatives (cfr Karlin/Rubin theorem).  But you are right that this is only in exceptional cases and certainly not in the setting for the Hosmer-Lemeshow test.
Comment: @fcoppens: Good point! You only get UMP tests by severely restricting the alternatives under consideration to values of a scalar parameter, & even then not always. Even considering whether a test's inadmissable - there's at least one other test that has greater power under all alternatives - would require restricting the alternatives too much for a general-purpose GOF test.


Correlation between parameter estimates in regression analysis

Question: I wonder what cause the correlation between parameter estimates in regression analysis?
Does misspecifacation of model cause such a correlation? Or does multicollinearity cause such a correlation?
In the existence of such correlations between parameter estimates, do we need to search the reason and study on the model to decrease such correlations?
I ask this in the sense of simple regression. But in real, I  work with a logit model with categorical dependent variables. And in my situation there is strong correlation between dummy varaible's(in the levels of a categorical variable) parameter estimates.
I will be very glad for any help.Thanks a lot.
Answer: The source of correlation between parameter estimates is the finite size of the design matrix.
Consider the OLS parameter covariance matrix estimate:
$$\operatorname{Var}[\, \hat\beta \mid X \,] = \sigma^2(X ^T X)^{-1}$$
The design matrix columns are usually correlated, that's normal, there's nothing wrong with it at all.
The formula is for finite sample, which is a very important consideration. Why? Because if you collect infinite number of observations, the diagonals of a term $(X'X)$ will become infinite, making uncertainty completely disappear. When there's no uncertainty, the question of the correlations is moot.
Of course, if the design matrix columns are uncorrelated this will also make parameter estimates uncorrelated, but this is truly a rare situation. Almost all
design matrices will have variables correlated.
Answer: In the simple regression, we assume the error terms are IID. The correlation between parameter estimates comes from collinearity of design matrix. and it is the only source of the correlation. If $X'X$ is diagonal, the parameter estimates are independent.


Filebrowser thumbnails "can not create img for private://"

Question: I am using Drupal 8.7.5 and PHP 7.3.7 running on Apache 2.4.6.
UPDATE: I'm trying to make available a library of images to the public. I chose the content type of 'Directory listing' to do this. In the Directory listing I have configured a private file path (i.e. located outside of Web root) to hold the images. Apparently I have done something right as it is working except for the thumbnails. I get constant 404 errors in my apache logs and the other errors through the web interface seen in the images below.
What have I missed in terms of getting the thumbnails working? Am I missing an binary or library to render the thumbnails? Did I not create or configure a path for the thumbnails?
Below are some of the steps I have done to configure the Directory listing.
UPDATE On the server I have installed Druapl 8 using composer and drush to $APACHE_CONTENT/drupal. The drush site-install then goes to $APACHE_CONTENT/drupal/web (drupal app_root) and have added a 'private' path as $APACHE_CONTENT/resources. This puts the 'private' path outside of the drupal site's DocumentRoot. The permissions and ownership for the 'private' path are 775 and apache.apache. Also, a .htaccess file was auto generated for the 'private' path. It contains the following:
# cat .htaccess
# Deny all requests from Apache 2.4+.
<IfModule mod_authz_core.c>
  Require all denied
</IfModule>

# Deny all requests from Apache 2.0-2.2.
<IfModule !mod_authz_core.c>
  Deny from all
</IfModule>

# Turn off all options we don't need.
Options -Indexes -ExecCGI -Includes -MultiViews

# Set the catch-all handler to prevent scripts from being executed.
SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
<Files *>
  # Override the handler again if we're run later in the evaluation list.
  SetHandler Drupal_Security_Do_Not_Remove_See_SA_2013_003
</Files>

# If we know how to do it safely, disable the PHP engine entirely.
<IfModule mod_php5.c>
  php_flag engine off
</IfModule>

In drupal/sites/default/settings.php I have:
$settings['file_private_path'] = $app_root . '/../../resources';

Viewing the Configuration->Media->File system is see:

I've configured a private file path to use with content created via
Content->Add Content->Directory listing.

I have then used scp to copy files into the private path.
Through the web interface, I can browse through the content but I never see the thumbnails. I get errors above my content.

In the apache logs I see 404s for the thumbnails:
GET /system/files/styles/thumbnail/private/Images/JPG/Marshal_Hub.jpg?itok=QwQds55m HTTP/1.1" 404 11730

I have GD installed.

While I can browse all of the content in the 'private' paths, none of the thumbnails get generated.
What do I need to do to get the thumbnails working?
Comment: Welcome to Drupal Answers :) What are you trying to do exactly? Making formerly public images private? Did you check private folder permission and ownership? Please edit your question for clarification. Many thanks
Comment: I have edited my original post but essentially I'm seeing a lot of 404 errors in my apache logs regarding the missing thumbnails for images I'm serving from a private file path for the Directory listing content type. The errors are like this: `GET /system/files/styles/thumbnail/private/Images/JPG/Marshal_Hub.jpg?itok=QwQds55m HTTP/1.1" 404 11730`. For a private file path, where do I create this path for the thumbnails?
Comment: I added a sub path to my 'private' path: 'files/styles/thumbnail/private/Images/JPG'. Now I get HTTP 403s: `"GET /system/files/styles/thumbnail/private/Images/JPG/Marshal_Hub.jpg?itok=dcxRfIFy HTTP/1.1" 403 22711`

Seems like it's the correct path but I'm just hacking things to see what works...still not working...
Comment: Did you update your Directory listings contents after copying the public folder to private? Or try adding a new content? If not, my guess would be that the path you gave in your settings.php is not the right one (hence the 404 errors because Drupal cannot create the thumbnails) and the images you are currently viewing are still the public ones.
Also when going to yoursite.com/admin/config/media/file-system, are you able to see your private folder path?
Comment: This is a fresh install of D8 done with composer and drush. I had no content in public. I set up private from scratch. So, no, I did not copy or update Directory listings. I get 404s only on HTTP GETs for thumbnails. All my content is listed and I can browse it through the web front end. Thumbnails fail to render and are 404. I manually added my best guess at a thumbnail path for the private path but it still fails to render thumbnails. I have uploaded new content but the thumbnail error persists in the Apache logs. Yes, I can see my private folder path @mysite/admin/config/media/file-sysyem
Answer: This is a workaround I used to solve the issue:
[https://www.drupal.org/project/private_files_download_permission]
I don't consider this 'magic' to be a fix. There is a lack of documentation and/or a bug in the filebrowser module.
How can one raise an issue with the developers?
PS: Also, now have another issue with the filebrowser module: sort order is not being honoured in Grid view!
Answer: It's my understanding that the 'private' directory can not be used to serve files. The whole point of this special directory is to provide a protected place for the files you choose to place in there. If the images are place in another directory under the 'sites/*/files' directory they should be accessible to be served.
Comment: Thanks. But I already can serve the files from 'private'. I just don't get thumbnails. So the issue is not that I can't serve the files. The issue is that for the files I'm serving, no thumbnails are generated. My understanding was that a 'private' directory is one to which you can't browse directly because it is outside of the web root. Also, 'private' paths can have access restrictions applied. See: [https://www.drupal.org/docs/8/core/modules/file/overview]


A confusing question related to three variables with fractions..

Question: Recently, I had a mock-test of a Mathematics Olympiad. There was a question which not only I but my friends too were not able to solve. The question goes like this:
If,
$$ \frac{1}{a} + \frac{1}{b} + \frac{1}{c} = \frac{1}{a+b+c} $$
Then what is the value of
$$ \frac{1}{a^5} + \frac{1}{b^5} + \frac{1}{c^5} $$
To, solve this question, I used a variety of ways like:
1) transposing variables in the first equation and,
2) putting a whole power of five to both the sides of the equation one. But, I was unable to find the solution.
The options were -- (a) 1 , (b) 0 , (c) $ \frac{1}{a^5 + b^5 + c^5} $ , (d) None of them.
So, I require any possible help. And, a complete answer would be most welcome. Thanks in advance.
Comment: Option $c$ looks good
Comment: Perhaps I am missing something, but surely the value of $ \frac{1}{a^5} + \frac{1}{b^5} + \frac{1}{c^5} $ would always be $c$, since $c$ is the same - $ \frac{1}{a^5} + \frac{1}{b^5} + \frac{1}{c^5} $?
Comment: Oh! I'm really sorry it was my fault, I wrote the wrong option by mistake. Option c is actually what I'm going to edit now. I'm really really sorry :(
Comment: Are you sure this is an Olympiad — the options?
Comment: @QuestionAsker well, this is **not an olympiad question** but a **mock test question**
Answer: Options $1$ and $2$ are wrong, take $a=1,b=-1,c=-1$.
Now notice given non-zero real number $a,b,c$ we have:
$\frac{1}{a}+\frac{1}{b}+\frac{1}{c}=\frac{1}{abc}\overbrace{\iff}^{\text{multiply by a+b+c}} 3+2(ab+bc+ac)=1\iff ab+bc+ac=-1$.
Analogously we have $\frac{1}{a^3}+\frac{1}{b^3}+\frac{1}{c^3}=\frac{1}{a^5+b^5+c^5}\iff a^5b^5+b^5c^5+a^5c^5=-1$.
Taking $a=2,b=2,c=\frac{-5}{4}$ satisfies $ab+bc+ac=-1$ but not $a^5b^5+b^5c^5+a^5c^5=-1$.
So the answer is $d$.
Comment: But, is there any way to prove it purely using algebra and not putting values and not doing hit-and-trial method. My argument for this is that, you are not provided with a statement like $ a, b, c $ can be any integer. Rather, their values are _probably_ well defined 'cause this statement is not in general case but a special case. This was what I was trying to do. Using pure algebra.
Comment: I'm sorry, but I don't see how $\frac{1}{a}+\frac{1}{b}+\frac{1}{c}=\frac{1}{abc}\overbrace{\iff}^{\text{multiply by a+b+c}} 3+2(ab+bc+ac)=1$. Is it $a + b + c$ before the $\iff$? Even so, how does the multiplication work?
Comment: multiply both sides of the equation by $a+b+c$, the right side becomes $1$ and the left side becomes $1+ab+ac+ba+1+bc+ca+cb+1$
Comment: The subset of $\mathbb R^3$ satisfying $\frac{1}{a}+\frac{1}{b}+\frac{1}{c}$ is a one sheeted paraboloid.  and the function defined as $\frac{1}{a^5}+\frac{1}{b^5}+\frac{1}{c^5}$ restricted to the paraboloid takes on a ton of values.
Answer: $\frac 1a + \frac 1b + \frac 1c = \frac 1{a+b+c}\\
\frac {ab +ac + bc}{abc}= \frac 1{a+b+c}\\
(a+b+c)(ab +ac + bc) = abc\\
a^2b + a^2c + ab^2 +ac^2 + b^2c + bc^2 + 2abc = 0\\
(a^2b + ab^2) + c(a^2 + b^2 + 2ab + cb + ca)= 0\\
ab (a+b) + c((a+b)^2 + c(a+b))= 0\\
(a+b)(ab + c(a+b) + c^2) = 0\\
(a+b)(a(b + c) + c (b+c)) = 0\\
(a+b)(b+c)(a + c) = 0$
$a+b = 0$ or $a+c = 0$ or $b+c =0$
Suppose $a=-b$
then
$\frac 1{a^5} + \frac 1{b^5} + \frac 1{c^5} = \frac 1{c^5}$
And going throught the other possibilities it becomes clear that c) is correct
Comment: Not true when $x$ is even.
Comment: From your result, I made an inference, please correct me if I'm wrong. That $ \frac {1}{a^x} +  \frac {1}{b^x} + \frac {1}{c^x} =  \frac {1}{a^x + b^x + c^x} $ if such a condition like the equation one is given.


Proving subset sums with the pigeon hole principle

Question: I'm having trouble with three problems dealing with the pigeon hole principle. They are:

Prove that any subset of size ten of the first 40 positive integers must have two different subsets
of size three with the same sum.
What is the largest value of m for which it is true that any subset of size ten of the first m positive integers must have two different subsets of size three with the same sum?
Show that any subset of size ten of the first 24 positive integers contains two pairs of values with the same sum.

My attempts:
For problem (1) I essentially tried to brute force it with a C++ program. I got some interesting results (if they turn out to be right). My program returned that there existed 9,880 possible sums for the first 40 positive integers (assuming size three) (e.g.) a+b+c and that there was only 112 unique sums. Therefore, by the pigeon hole principle there must be a sum which repeats.
Problem (2) I honestly did not know how to deal with this problem. However, when I first read the question I did not think that m had an upper bound (i.e. it works for every m). But, now I'm not so sure and do not know where to start.
Problem (3) For this problem I also tried a brute force method and got a total of 176513040 possible sums with 47 being unique. However, I don't think this is right. Nor do I think that my answer to problem 1 is right.
Can anyone offer some advice?
Thank you!
Answer: It's hard to think about questions such as these because there's lots of quantifiers. So here's a special case of question 1:

Show that the set $S = \{1, 3, 4, 7, 9, 10, 11, 13, 16, 19\}$ must have two three-element subsets with the same sum.

Now that the question is this specific, we can of course solve the problem by inspection: $\{1,10,19\}$ and $\{9,10,11\}$ have the same sum, for example. But we also want to generalize this to other sets eventually.
So here's an argument that generalizes: there are $\binom{10}{3} = \frac{10\cdot 9\cdot 8}{6} = 120$ different three-element subsets of $S$. The smallest possible sum is $1+3+4 = 8$ and the largest possible sum is $13+16+19=48$, so all three-element subsets have a sum between $8$ and $48$. Since this leaves only $41$ possibilities for the sum, and there's $120$ three-element subsets, some two of those subsets must have the same sum.
So now, to find answers to your questions, we need to:

Show that the same argument holds when $S$ is any ten-element subset of $\{1, 2, 3, \dots, 40\}$. We don't know what the smallest and largest possible sum are in $S$ without knowing $S$; but how small and how big can they get?
Once you've figured out the answer to question 1, for what values of $m$ in place of $40$ would the same argument work? (How does the range of possible sums depend on the value of $40$?)
Can you make the same argument work with two-element subsets instead of three-element subsets?
Comment: Thank you very much! I seem to have gotten the first two answers. However, I am still stuck on the third. If I use the same argument from problems one and two then we end up with more sums then there are subsets which is a contradiction. How would I go about answering (3)?
Comment: That one is actually a bit tricky; you get $\binom{10}{2} = 45$ possible subsets, and also $45$ possible sums (ranging from $3$ to $47$), so pigeonhole doesn't quite apply. However, if you think about what a set $S$ would look like if both $3$ and $47$ were possible, you can find two pairs with the same sum in another way.
Comment: I'm a bit stuck. The only other way I can think of would be the generalized version of the pigeonhole principle, but I'm not sure it works for this problem. Is there anything else you would suggest?
Comment: What elements must $S$ contain if it has a pair summing to $3$ and a pair summing to $47$?
Comment: $S$ would need to contain $\{1,2,...,46\}$ (assuming that we can use (1,2) and (1,46) as pairs). If not $\{1,2,3,...,44\}$ also works.
Comment: $S$ is a subset of $\{1, 2, \dots, 24\}$. It can't contain $46$ or $44$.
Comment: Oh right. $S$ needs to contain $\{1,2,...,23,24\}$. Taking $(1,2)$ and $(23,24)$ as pairs.
Answer: What you need to do is to correctly count the possibilities.
For problem (1) you need to know how many subsets of size three there is of a set of size 10. To get a subset you will pick three different elements, the first element can be choosen in 10 ways, next in 9 ways (since the first picked element can't be picked again) and finally the third in 8 different ways. This gives $10\times 9\times 8$ possibilities, but it doesn't matter which order you pick the same elements, given a pick there's $6=3\times 2\times 1$ (which is seen in similar way) ways to reorder that which brings us to $120={10\times9\times8\over3\times2\times1}$. Then you know that the certainly can't be larger than $120=3\times40$ and no less than $3=3\times1$ (yes we can tighten somewhat, but we need not).
Note well that the error you did in your attempt on the first question was that you tried to pick subsets of the entire set, but the question asked about a subset of size ten of that set. Your number is however correct (check with formula below), but be careful about reading the question.
The other is solved in similar way. The formula you need is that the number of subsets of size $k$ out of a set of size $n$ is $${n!\over k!(n-k)!} = {n\times(n-1)\times\dots\times(n-k+1)\over k\times(k-1)\times\cdots\times1}$$
The notation $n!$ means the product of all positive integers up to including $n$. We can see that this is the number of ways we can reorder a collection of size $n$. We see this by the reasoning above, to arrange the collection we sequentially picks $n$ elements, first we can pick $n$, then there's $n-1$ left so we can pick $n-1$ and so on until we only have one left. One can interpret the formula as a count of the number of ways we can rearrange the whole set (of size $n$), the subset (of size $k$) and the rest of the original set (of size $n-k$).


Trying to pull certain data pieces of an LRS record, need catch-try assistance

Question: I have a script that is 90% complete and working. I'm using the older code for getting LRS statements. It is the one that uses tincan.js. I am trying to pull down and parse the following: Actor, verb, object, timestamp and the id under contextActivities. It is the last one that is giving me fits. Could you please look at my last catch/try statement and tell me where I'm going wrong? Thank you.
<!DOCTYPE  html>

        <meta charset='UTF-8'>
        <title>Get my Statements</title>

Get statements  101

Answer: You are trying to access a property that will never exist:
statements[i].target.contextActivities
The last portion of that will exist in statements[i].context when it exists, which is not always unless you know the statement ahead of time. Pretty much all of the properties should be checked for null before accessing them, doing so will allow you to remove your try/catch blocks. The property names themselves are very stable and the library explicitly assigns null values to known properties. Additionally the other property will hold an array of Activities (when it is populated) so you should access them by index, so contextActivities.other[0].id etc.
For the target property you should check that it is an instance of a particular class as it can contain multiple types of objects (so can actor but they effectively match). In general for this specific task you may also want to leverage the toString methods where possible, they do very similar work.
Comment: Thanks much. I will give it a try.


using entity reference fetch different fields automatically

Question: using entity reference, i am able to relate to nodes with one another. Thats great. The user while entering their blog has an option to relate it to nodes of my custom content type. This works great. Now i want that once the user has selected the node to be related to, the blog should not only show the title of the n node but also other fields from that referenced node. How can i achieve this?
Regards
Answer: On the content type that has the reference-field, on the "Manage Display" section, you can choose if you want "Label" or "Rendered entity", which will output the full node.
Comment: hey thanks a lot...just to add on..in the referenced node i have many fields, is it possible to just get a few fields only and let say trim the length and get the image and say just one more field and ignoring the others...i hope you understand my use case...if not just tell me i will try to be more clear
Comment: I understand your doubt!
You can set display to 'hidden' . The referenced node will still be loaded, and you can access its variables and print them as you wish. If you want more detail, just use  on your node template , and work from there.


Projectile one dimensional optimization problem

Question: Suppose a projectile is thrown with initial velocity $v$ aimed at some angle $\alpha$ wrt ground to hit a target of some height $h$. Given the horizontal distance satisfies (g = 10 m/s)
$$ \left( \frac{ g }{ 2 v^2 \cos^2 \alpha } \right) x^2 - (\tan \alpha ) x + h = 0 $$
My question is. This equation is supposed to give the horizontal distance, but it has two roots. I know that $x > 0$ and we can disregard the negative root. But, how about if for some $v, \alpha ,h$ we obtain two roots $x_1$ and $x_2$ which are both positive. Which one is supposed to be the horizontal distance we want?
Comment: If $\tan \alpha >0$ and $h>0$ there will in fact be 2 positive solutions for $x.$
Answer: Your projectile will reach height $h$ twice during its flight (assuming $h>0$), once on its way up and once on its way down.
If there are no other details then both $x_1$ and $x_2$ are valid answers to the question 'When will the projectile reach height $h$?'
Comment: Contradicting Aristotle, who wrote that a thrown projectile travels in a straight line until at some point it drops straight to the ground. ("You can't make this stuff up.")
Answer: If you get two solutions both will be correct.  One will be very fast, traveling on almost a straight line to the target.  The other will be a high arc also hitting the target.