Finding the optimal value of k in the k-nearest-neighbor classifier: is this cross-validation?

Question: I have collected 1000 data points with each data point belonging to eight categories. I would like to be able to correctly estimate the categories of any new data by using the k-nearest-neighbor classifier. This requires me to find the best value of the hyperparameter k. What I would like to do is to try various values of k, maybe from 1 to 40, then take every data point that I have (because why not use them all?) and see if it is correctly classified using the given value of k. Then for each k I will get an overall error proportion (how many of my data points were correctly classified) and I will use the k that gave the lowest error. Is this a reasonable way to do this? What is this method called; is it cross-validation where the size of each verification data set is equal to one? Is that the same as leave-one-out cross-validation?
Answer: That's almost right, but there's one extra step needed, which is to fit the model and choose $k$ using different sets of data (fitting in the case of knn being just remembering all the data points). If you fit the model and select the hyperparameters on the same set, the error will underestimate the true generalization error, which is the error you'd see on new data drawn from the same distribution. This means the classifier may do worse on new data than you'd expect. Cross validation is a way to choose $k$ that tries to minimize the generalization error.
Here's how to select $k$ using $d$-fold cross validation. It's usually called $k$-fold, but we're using $k$ to talk about the number of neighbors.

Split the data into $d$ disjoint, similarly-sized subsets
Hold out the first set. This is called the validation set.
Train your classifier on the remaining data. In the case of knn classification, just remember all the data.
For each value of $k$:

Classify each point in the validation set, using its $k$ nearest neighbors in the training set
Record the error

Repeat steps 1-4 for all $d$ choices of the validation set.
For each choice of $k$, find the average error across validation sets. Choose the value of $k$ with the lowest error.
Construct a final classifier using all of the original data and the chosen value of $k$. This is what you'd use to classify new points.

In the case where $d$ is equal to the number of data points (i.e. each validation set contains a single point), this is called leave-one-out cross validation.
If you want to estimate the generalization error of the final classifier, some extra work is needed. You can't just take the error on the validation sets, because $k$ was chosen to minimize this, so it's an underestimate of the true generalization error. Another way of thinking about this is that procedure for choosing $k$ has to be considered part of the learning algorithm. The simplest thing to do is test the final classifier on a separate set of data you held out at the beginning (called the test set). This is a reasonable option if you have lots of data and a slow algorithm. Another option that takes longer but uses the data more efficiently is to use cross validation again. Here, there will be an 'outer' cross validation loop, where the held-out data is called the test set. At each step, you further split the data into training and validation sets and run a nested, 'inner' cross validation loop to choose $k$, as above. The final generalization error is estimated by classifying the test sets and averaging their errors.
Answer: $k$ nearest neighbors isn't exactly "trained" per-se, in that what you see is what you get. However it does explicitly depend on where your data is and how clustered it is. Areas with a lot of different features (say, binary) will give you ambiguous results. Leaving data out will give you some sense of the variance you would expect to see when you end up adding more data later on. So this is more of a grid-search than bonafide cross validation.
As an example, imagine your points are coming from some spatial distribution (along with some binary features), for example clustering in various areas spatially (but not necessarily feature-wise). In this case looking at random subsets of your data would give you some impression of where your nearest neighbor algorithm is doing the best and worst, with respect to the expected number of points in a certain cluster.
Comment: Okay, thanks. But doesn't it help more to consider the whole data rather than a random subset of the data in order to find the best value of *k*? I mean, doing that gives me more information, right?
Comment: Also, I understand now that this is grid-search, which is quite easy to do here because I only have one hyperparameter, but I am not quite sure what that has to do with the comparison between using a limited number of data points and the full data set. Could you please clarify?
Comment: If you are restricting yourself to just your data, and don't anticipate to get more data in the future, then perhaps considering your full data set is reasonable.
Comment: I do expect to get more data in the future though. Should I restrict myself to a part of the dataset in order to prevent overtraining? And if I do that, would this method then be considered to be cross-validation?


Installing "duplicate" on SSH /home/user

Question: I am on a SSH server and want to use a program that is at /usr/local/bin but I don't have permission.
So I copied the program installation files from my system to my user on the server in order to install it at /home/user.It is probably a different release of the program than the one installed at /usr/local/bin.
Is there a possibility that I screw up the way the already installed program works?I'm guessing not,because I am not the administrator so it would stop me from messing something up.
Answer: It's very unlikely that you can screw up the existing installation. No locally "installed" program will be setuid, so it will run with your permissions, meaning that it can't do anything you can't do with other programs.
But if you just copy the binaries from your own system, they might not work as intended. Worst case is that it doesn't work at all (would e.g. be the case if the architectures are different) more likely it will depend on some library that isn't an appropriate version. The best way to get a program that will work, is to compile it yourself, and as a part of that configure it to use local versions of all libraries that you need and can't rely on being (and remaining) in a good version on the system, of course you will probably need to compile a lot of libraries yourself.
In short: You need to be really sure before you start this, because it quickly becomes a huge project. Have you tried asking the administrator if they could change the permissions so you can use the installed version?
Answer: If you don't have Admin/Sudoer access, the likelyhood of you breaking your system (for all users) is probably fairly slim, then again, it's possible that your system has been customized as such to allow a non-sudoer to install programs, or make other harmful modifications. It's also possible, that installing a duplicate for only your local user could cause problems for just your profile.
My best advice would be to proceed with caution, and keep track of your changes in the event you need to back them out. If you're worried about breaking the system for other users/user profiles, you're probably not going to do so. The community may be able to provide more clarity with a more detailed question eg: What distro? Have many modifications to the system been made? What application are you trying to install... etc.


Is it a good idea to inherit from standard widgets and set my own default values in C#?

Question: When designing app in Visual Studio (C#), if I know that I will have a certain number of DataGridViews that all have same properties (like width, height, color, some other properties like: disable option to directly edit rows, etc.) is it ok to make my own class ("myDataGridView") that inherits DataGridView class and make all adjustments there, and then just instantiate that class later in code? Like this:
<code>//my class:
class myDataGridView : DataGridView
{
    this.BorderStyle = <someValue>
    this.ColumnCount = <someValue>
    //etc.

    public void method1()
    {
    //some code...
    }
    public void method2()
    {
    //some code...
    }
}

//instantiate it somewhere:
myDataGridView dgv1 = new myDataGridView();
myDataGridView dgv2 = new myDataGridView();
myDataGridView dgv3 = new myDataGridView();
</code>
Is that ok regarding OO principles? My friend says that putting code like
<code>this.BorderStyle = <someValue>
</code>
in myDataGridView class is bad practice because adjusting properties like that will overrun properties of dataGridView which some other developer may adjusted visually in Visual Studio, if you know what I mean. Does that matter? I mean, if I want to treat my DataGridView as an object then it can have its properties and behavior, right? And having code that adjusts properties of DataGridView in my class is ok, readable, and every other developer who wants to change some properties can change it in myDataGridView class. It that kind of practice bad or wrong? If it is, what is the best practice when you know your app will have many DataGridViews which have same properties/behaviour? Thank you.
Answer: In my opinion, this is not the right approach. Inheriting controls just for few property modifications does not make sense. You're not introducing any additional behavior in the control, Rather,try writing some initialization helper methods to do this.
Comment: If you could tell us more about your objective, then fellow developers can come up with a good design idea for you.
Comment: Ok, for example, I have four data tables in my SQL database. For every table I have one DataGridView. I wanted to make myDataGridView class that inherits DataGridView class (here are all adjustments of properties which need to be same for every DataGridView), and then I wanted to make four more classes (myDataGridViewTable1, myDataGridViewTable2...) that inherit that class (myDataGridView). Each of those four classes would have their own behaviour, like update(), delete(), get(), etc. Those are using stored procedures in database. So, I could use, for example, dataGridView1.update();
Answer: If you are worried that your modifications to the properties get overwritten by a visual setting by the developer you may hide them using [Browsable(false)] attribute.
However, from OOP perspective inheritance is not encouraged unless really necessary. There is a golden rule : "Favor composition over inheritance". As this. __curious_geek said it's not a good idea to inherit controls to set properties this way but it's good practice to always have common base class for your business classes and screens. You may make a function in the base class of all screens to properly adjust the grids for you the way you want. If you make it virtual, you may override it in derived classes to customize the behavior per case as needed.


Do these pipes need to be winterized? How?

Question: I already winterized the sprinkler system itself and I am wondering if I need to winterize these copper pipes as well. Should I open up the bottom plug to drain the verticle pipes? I live in the Northeast of the USA. I don't have a bleeding valve inside the house that's close to these outside pipes.
Comment: the shutoff should be inside the house .... turn it off and drain all water from the copper pipes
Comment: @jsotola I already turned off the shutoff inside the house. Should I open up the plug at the bottom? I think there is still water in the verticle pipes. This is not a very efficient design if I have to open the plug every year.
Comment: You could replace the plug fitting with a ball valve, but for something you only need to open once a year, this is a perfectly functional design.
Comment: `Should I open up the plug at the bottom?` ... why are you asking this? ..... i am 100% certain that you know how water behaves, therefore the answer should be obvious to you .... do whatever you have to do to empty the pipes
Answer: Yes, you need to winterize them. That looks like a backflow-prevention device, which prevents water from the lawn from flowing back into the city pipes (in the event of a loss of pressure). Typically, when I lived in TX, my winterization procedure was to shut the inlet valve (the green-handled valve on the right side of the device in your picture) and turn the screws on the short stubby pieces 1/2 or 1/4 turn and drain the water from the backflow device itself. Turn the outlet valve (green-handled valve on the left of the device) partway closed. Wrap the piping in foam insulation, and possibly wrap the backflow prevention device itself in a towel or shirt for the few nights it dipped below freezing.
Given that you're in the Northeast US, though, you should have a shutoff valve / drain somewhere inside the house. Any piping prior to the inlet valve will still have water in it unless it's drained, and that portion of the pipe from the house to the inlet valve will freeze if it isn't drained. Once you find the shutoff, open its drain if it has one, and open the drain on the short vertical piece below the backflow device. Once all the water has drained you can close the drain on the short vertical piece. In the spring, close all the drains and open all the valves.
It may be a poor design, but the important thing is to make sure that any pipes exposed to the exterior are not full (or close to full) of water. The only way to do that is to drain the water in the system as part of the winterization.
Comment: Those side plugs on the valves are for bleeding.  They have added full and proper *valves* (operated by a screwdriver) and what look like compression fittings that would be very suitable for compressed air!  That would apply to the top valve, the side valve may be merely to allow in air when it is drained from somewhere in the house interior.
Comment: I'm very disappointed with the plumbing job for the sprinkler system. The inside shutoff for the irrigation system is very far from the outside pipes (more than 20 ft) and between the outside and the inside shutoff there is not a bleeding valve.
Comment: If there was a bleeding valve inside the house, the whole thing would make perfect sense. But there is not. Opening up the bottom plug seems the only way. I may replace it with a ball point valve for easier future operations.


Does Revelation self-consciously bring a close to the canon?

Question: The Revelation of John is placed in a lot of bibles at the end of the canon. This has always seemed natural to me, but on reflection perhaps it is simply an artifact of the books uniqueness as an Apocalypse in the New Testament canon. Are there reasons, though, to think that John might have self-consciously written it as a close to the Scriptural canon (however he might have construed it)?
Answer: Revelations is certainly the last book of the bible; however, canonicity wasn't resolved concerning the New Testament until the Council of Trent in 1546; in which it was established that 27 books which we refer to as New Testament were confirmed as "Articles of Faith", and "anathema" was ascribed to those that rejected them.(Reference: Canon of Trent)
What many have mistakenly inferred Revelations to be the last book of the bible comes from Rev. 22:18-19 (KJV),

For I testify unto every man that heareth the words of the prophecy of
  this book, If any man shall add unto these things, God shall add unto
  him the plagues that are written in this book: 19 And if any man shall
  take away from the words of the book of this prophecy, God shall take
  away his part out of the book of life, and out of the holy city, and
  from the things which are written in this book.

Various preachers have interpreted "this book" to mean all the books of the bible, when in fact John is only referring to 'his book': the "tou" or "toutou"(the/this) are in the Genitive Neutral Singular, which refers only to his book, not all 'the book(s)' which comprise the bible.
Another reference to the same issue is when John describes the "adding of plagues" to those who would "add" to his words. These plagues are spelled out specifically in Revelations, whereas one would have to 'conjure' those from reading the rest of the bible.
Whether or not John saw Revelations as being the "last book" is speculative; but Jesus Himself said,(Mk. 9:1 KJV)

"And he said unto them, Verily I say unto you, That there be some of
  them that stand here, which shall not taste of death, till they have
  seen the kingdom of God come with power."

John certainly 'saw' the Kingdom of God come in power, and Jesus returning to reign on earth. This became a point of controversy which is described in John 21:23,(KJV)

Then went this saying abroad among the brethren, that that disciple
  should not die: yet Jesus said not unto him, He shall not die; but, If
  I will that he tarry till I come, what is that to thee?

John saw and understood the "Alpha and Omega"(Rev. 1:8-KJV)

I am Alpha and Omega, the beginning and the ending, saith the Lord,
  which is, and which was, and which is to come, the Almighty.

who is the beginning and end of all Revelation. It was his gospel where Jesus says,

"Jesus saith unto him, I am the way, the truth, and the life: no man
  cometh unto the Father, but by me."(John 14:6-KJV)

John knew that all truth was contained in Jesus; and seeing the end in the New Jerusalem coming down from heaven(Rev. 21:2), he saw "God shall dwell with His people".(vs 3). So while it is highly speculative that John imagined he wrote the last book, it is no stretch to say that John saw the final day, in which those who were redeemed by God from all humanity would dwell with Him for all eternity.
Answer: Once the New Testament books were placed together in one volume, it was natural to place the gospels first, followed by all the epistles and then place Revelation last, whether or not we think of Revelation as a prophecy of the end times.
John would not have known what books of the New Testament were yet to be written; in fact there was no thought given to defining a New Testament canon until after Marcion had defined his own canon in the middle of the second century, in competition to the centrist Church.
It is only as codices began to replace scrolls that it became feasible to even think of placing all the books in the one codex. Christians began to use codices around the end of the first century, at about the time John of Patmos is thought to have written Revelation, but he can not be expected to have considered that the Church would one day define a canon of scriptures, bound in a single volume, with the Apocalypse last.
Answer: Tau's answer ends with this note:

So while it is highly speculative that John imagined he wrote the last book, it is no stretch to say that John saw the final day, in which those who were redeemed by God from all humanity would dwell with Him for all eternity.

Theologically this is surely the most salient point that can be made—whether or not John self consciously knew the work he just penned would end up being the last in the Scriptural canon or not is nearly irrelevant to the meaning of the work itself.
That being said I don't think it's much of a stretch to speculate that he was aware of this detail. Speculative? Sure, but not highly so, and not even that much of a stretch.
John refers to himself as "The Elder". Many commentators have noted that this could have more than one connotation. Sure the position he held in the church could be labeled as the office of Elder, but at the same time John likely had a second meaning in mind. By this time he was an old man. He had more gray hairs than anybody else on the team.
As a matter of fact, all the rest of the team was dead. As a mere youth at the time of Jesus' ministry and having been blessed with a generously long life, he was the last living member of the original Apostles. Even Paul had been pushing up daisies for nearly 30 years!
Revelation was likely the last book physically penned. The most probable candidates for later books would be John's own three shorter letters: which leaves John still having a pretty good idea of where  his work fits in. The other NT books all date to 15–50 years earlier. Some have proposed a late date for Mathew; yet if the book is original material the earlier dates make more sense and if it is a compound work refactoring existing material then a late date for its authorship isn't really a case  for adding new material that John wouldn't have been aware of.
Given his location in Ephesus up until the years directly preceding his exile on Patmos (where he likely wrote Revelation) he would have had access to nearly all the scattered documents that later came to be explicitly known as the canon. Ephesus was a hub through which all the letters as well as of course the gospel texts would have eventually made their way. John's position in the church there would have guaranteed that he read them all. He surely had a pretty good handle on what texts were out there, what had been written by his fellow Apostles, what Paul had penned, etc.
And as an old man he knew that there wasn't anybody left that could give the kind of witness he could.
Whether he had in mind the kind of organized canon we have today could be debated. As others have noted his own warnings about not adding to or taking away from the book is likely a reference to either the one book or the scope of the Apostles' teachings in general rather than inclusion or exclusion from a canon—but I think it's safe to speculate that John did in fact have some realization of where his work fit in the context of the rest of Scripture.
Answer: Does Revelation self-consciously bring a close to the canon? The question seems to presume that the scroll has a consciousness, which is not likely what was meant so I'm interpreting the question to be, "Does the author of Revelation intend that Revelation be the last scripture to be written?"
I will presume that this question is based on the blessing and curse that ends the scroll:

Rev. 22:18-19 For I testify unto every man that hears the words of the
  prophecy of this book, If any man shall add unto these things, God
  shall add unto him the plagues that are written in this book: 19 And
  if any man shall take away from the words of the book of this
  prophecy, God shall take away his part out of the book of life, and
  out of the holy city, and from the things which are written in this
  book.

By placing this blessing/curse at the end of his scroll he clearly intended his own work to be "canon" or "rule" that was to be taken as divine in authority.
Since Revelation was written on a scroll and not at the end of a codex (book) that contained other writings it can safely be asserted that there is no reason to think that the author would be intending to canonize any and all text that would then be bound up together with it in the future. Had that been his intent then the contents of Sinaiticus would have thus been "canonized."
So why does it appear last in Protestant "Bibles"?
First of all, "canonization of Bibles" is not a scriptural concept. The scriptures never speak of a "Bible". Instead it speaks "writings" ("scriptures") and authorizes them all as "god-breathed":

2Ti_3:16  All scripture is given by inspiration of God, and is
  profitable for doctrine, for reproof, for correction, for instruction
  in righteousness:

And it speaks of "sacred literature":

2Ti 3:15  and how from childhood you have been acquainted with the
  sacred literature, which are able to make you wise for salvation through faith in Christ Jesus.

However many religions have the concept of a "canon". The Catholic canon that has prevailed for most of history was a Latin text called "The Vulgate" which was authorized by the Pope to be used in the Catholic Church. It contained several texts that do not appear in the Protestant canon. Other groups, such as the Mormons have other texts in their canon.
The Revelation came to be placed at the end of the Protestant canon because Martin Luther produced a German language Bible and he placed all of the texts from the OT that he considered bogus at the end of the OT (now called "The Apocrypha) and all the texts from the NT that he considered bogus at the end of the NT. Luther considered Revelation, while of interest to Christians as literature, to be the ravings of a mad man so he stuck it at the end.
So, depending on your faith:

the organization of the texts of the scriptures in a book is post-revelation affair of no religious importance
Revelation is non-canonical and so stuck on the end of the revealed books
Revelation is canonical but so are the books of Mormon
only tradition recognizes the concept of a canon
Revelation, by divine inspiration, closes the canon which, now by divine decision is the Protestant "The Bible"

Personally I consider the concept of "The Bible" to be a completely bogus one as the scriptures themselves say.
For more information:
https://en.wikipedia.org/wiki/Luther%27s_canon
Comment: This interpretation of 2nd Timothy 3:16 as evidence against a limited set of writings being considered god-breathed is really poor hermeneutics and miss-informs the rest of this answer.
Comment: @Caleb  What principle of hermeneutics did I violate? And if "The Bible" exists, where in time and space is it? Is it the Vulgate? Luther's Bible? The KJV 1611? KJV 1640? The Eastern "canon"? Where can I view "The Bible"? Is it in a museum in London? Chicago? Vaticanus? Sinaiticus? Is it infallible? I'd love to see it.


What is the best way in terms of non blocking table for creating NONCLUSTERED INDEX for partitioned tables?

Question: We are using SQL Server 2016 and have 3 partitioned tables (every table around 300-500 milion of records). For every table would like to create NONCLUSTERED PARTITIONED INDEX. What is the best way to avoid blocking table in process of creating each NONCLUSTERED PARTITIONED INDEX. Could we use ONLINE option for this kind of indexes ?
Answer:
Could we use ONLINE option for this kind of indexes ?

Maybe. The documentation states:

Online index operations are not available in every edition of
Microsoft SQL Server. For a list of features that are supported by the
editions of SQL Server, see Editions and Supported Features for SQL
Server 2016.

The reference shows online indexing requires Enterprise Edition.
Comment: Also note that `ONLINE=ON` does not mean there will be no blocking, it just reduces the amount of blocking fairly significantly. Additionally, keep in mind that even with minimal blocking, depending on how resource intensive the index creation is and how your server is provisioned, it could still cause global resource contention and affect the performance of your server for other queries. So you may want to consider a maintenance window, if possible, as well.
Comment: Thank you. Just to confirm - The only option to avoid blocking is ONLINE option correct ?
Comment: @grochu, that is correct. You'll need to plan a maintenance window if `ONLINE=ON` isn't feasible in your environment.
Answer: Be aware that if your system is very busy the ONLINE option can take a very long time to run.  BTW don't forget to partition align the index.


Tiny amount of USDT returns during swap

Question: This transaction is swapped USDT -> USDC. Also in this tx, I see a transfer of 0.000002 USDT to the user's address. The question is what this transfer means and why it happens?
Answer: Probably a rounding leftover.
USDC uses 6 decimal point precision, so 0.000002 would be a reasonable division leftover.

Tiny amount of USDT returns during swap


How to link a shapefile with points to a shapefile with lines?

Question: I have a shapefile with points (nodes) and a shapefile with lines (pipeline) which I have extracted from a water distribution network. I have been able to transform the points into the right projection and the layer looks good, but the lines I dont know how to project or illustrate. In the attribute table there is the name of the start node and end node. Can I link the shapefile with lines to the points?
Comment: Something for the future - You can edit your question to reference ArcGIS rather than needing a comment.
Comment: Its in ArcGIS by the way
Comment: If you do a Join, you can link the line to its start or end point but not both (assuming the point names are the same as the start and end node names in the table). I'm not sure what issue you have with projecting or 'illustrating' the lines - if you add the shapefile the lines and points should line up if you do the same transformations on both. We might be able to provide better direction if you can edit your question to explain more what you're trying to do (what does this 'link' accomplish?).
Answer: Join the tables together.  Right Click on the point shapefile and join it with the line shapefile.
Answer: You can do Spatial Adjustment (Edge Snap/Edge match) to link your points and polyline. I have been doing this as well as I'm working with water utilities. Just make sure you are creating a link for the right point to the right node in the polyline. A few links will do.


How would an alien civilization equivalent in advancement to humans today hide their existence from future humans?

Question: 160 light-years from Earth, 1000 years in the future, exists an alien civilization roughly equal in its technological advancement to modern civilization on earth. They fear the possibility of the expanding humans invading and destroying them. Future humans have already colonized almost all neighboring star systems. Surely humans would detect their radio emissions, not to mention that an expedition made to their home system would likely occur in the near future. How would these aliens avoid detection?
Comment: Welcome cxnhgfh. Could you tell us about your aliens, how they detected such an advanced humanity without being detected themselves? What sort of star/system they live in? Do they have nuclear power, space-flight, satellites orbiting? What's the atmosphere on their planet, its gravity/size? Is there abundant water in any form? Also, it's tricky to know what a 1000 year advanced humanity  will look for in a planet or system for resources or colonisation, can you tell us what you've decided there?
Answer: It's not the Radio Transmissions that I'd be worried about
It's a planet that is capable of sustaining life. And that is probably the biggest hurdle to overcome. If Humans have Inter-stellar tech and the Aliens don't - then they are probably mostly out of luck...
There's a number of problems:

The Aliens don't know what the primary means of gathering information that Humans use. You mention Radio Waves - Do you humans use Radio Waves anymore? Or is that such an old-fashioned technology that it is considered Quaint and a Novelty?
as per my Title - I'd be more worried about a Human Surveyor going 'Mmmm Liquid water, breathable atmosphere, green plant life?!? Bust out the Flags boys, it's colonizing time!' - and that is something that we currently could not mask at a planetary level.
Too many Unknowns.

However... There's a few scenarios where we can somewhat address those questions in a plausible-ish manner, which allows us to fudge Technology with a bit of 'necessity' handwavium.
Option 1: You mentioned other planets were Colonized - if they also had intelligent, space-fairing life, then some of those Aliens could escape, warn your Planet about the Humans and what their capabilities are - If it's Radio Waves - then radio transmissions above a certain wattage are now illegal on pain of death, everything runs in Cables. Then, high above the Atmosphere, they let off a large number of Nuclear explosions.
The Humans observe the following: Regular radio traffic, indicating intelligent life, a mass of Nuclear explosions and a sudden cessation of all Radio Waves - they presume that the entire planet was irradiated in a global Nuclear war - and decide to give it a miss, not worth the effort.
Option 2: Human Nomads/Hermits - especially someone who was high in the Military/Colonization efforts and got disillusioned with the whole thing and sort out a place to be alone and die - met the Aliens and told them how they selected Planets. This then enables them to try and either fake signals or gives them a means to boost their technology up to the point they can disguise the habitability of the planet: perhaps sending an artificial signal to fool the sensors, perhaps good ol' fashioned camoflage on a global scale - dying the Oceans, putting netting over the crops etc.
Answer: To what length will they go?
There's a limit to this question, and to demonstrate that let me change the conditions a bit. If we ask how a medieval society could hide from today's tech the answer is, "they can't." Most of what we do today would, at least initially, look like magic to them. Even after they grew accustomed to our tech, they wouldn't understand it. As a world-wide population, they simply wouldn't know where to start to hid from (e.g.) a satellite.1
What makes more sense is asking how a newly post-industrial civilization (e.g., England about 1895) would avoid identification as a post-industrial society.  Now we're talking about early radio, coal-fired pollution, railroad tracks, and a sudden increase in building sizes. Easy to see. Easy to detect. To be fair, even they would have the devil of a time hiding their very existence from we modern humans, but with some ingenuity (which they had), they could perhaps hide their tech level.
What do those examples suggest to us? You have two problems:

As you've mentioned, your society has emissions. Principally electromagnetic, but also in terms of pollution and putting a significant amount of stuff into orbit. You also have considerable ground construction and ground-level light pollution. Just to name a few. How would such an advanced people hide? By ridding themselves of those emissions — green activism on steroids. OK, not steroids... PCP.

But you also have people. There were large ancient cities... but not by today's standards. From the one data point we know about (humanity) we believe that there must be advancements in agriculture, medicine, sanitation, transportation, and other sciences to achieve seriously outrageous populations. So you could convince everyone to become vegans and live in tents, but the passing humans would inevitably think to themselves, "that's a honking lot of people living in tents and eating lettuce... I wonder how they're farming and moving all the lettuce... and where does all the crapola go?"

And, just to make a point, there's likely a third problem that we're simply going to ignore.

Humanity has been throwing stuff out into space (emissions, satellites, etc.) for a long time — and we have yet to find evidence of another species.2 We're going to simply ignore this. Sometimes it's what you gotta do.

So, what could you do?
We have aliens who, for the sake of argument, act and think a lot like Humans with circa 2023 technology who got very lucky and happened to detect the wake of a passing relativistic craft at the very moment that no one on the craft cared a whit about looking at their planet. Cheers! The knowledge that they're not alone in the universe and, much to their chagrin, they also aren't top-dog on the technology pyramid has driven the more military-minded among them to take over the world's governments and act, as they say, for the preservation of their race. And they decided to do one or more of the following:

Strict population controls are put in place to permit decentralization of core population groups. This isn't unbelievable as we not only have advocates of such measures today but also have governments who have implemented such measures. [1], [2]

Development of thermal-based energy production. Geothermal comes to mind, but the goal here is to develop energy production that doesn't require (a) large surface structures (e.g., dams) or (b) substantial pollution (e.g., coal or nuclear).

Development of dense hydroponic food industries. The goal here is to get away from large surface agriculture to avoid not only large trivially discernable tracts of farmed/ranched land, but also large irrigation projects.

A serious focus on recycling.

And why do you need all this?

Because the only way to hide from such an advanced civilization is to move deep underground and hope that civilization doesn't look at your depopulated surface and think to themselves, "hey, there's an upopulated and unclaimed paradise down there!"

From a story standpoint you'll need to ignore a lot of things because unless your universe is unusually repleat with habitable planets, the fact that you have a habitable planet will be the single most difficult thing to hide... unless you're willing to destroy the surface. If not, every passing high-tech ship will be want to take a very close look at the planet.

1 Let's ignore that there were a few thinkers who could imagine the fundamentals of the technology and, therefore, comprehend at least the basics of how they operate. They may not understand the mathematics of electromagnetic propagation... but they'd comprehend the basic operation of light. Random genius can get in the way of any worldbuilding effort... or be the convenient joker card that ensures success.
2 We could point to the "Wow!" signal as possible evidence of extraterrestrial life. However, we humans are somewhat paranoid critters, evidenced by our penchant for conspiracy theories, and yet that signal has not driven us to hide from our potential alien overlords. Consequently, I suspect a believable story would require better proof than the "Wow!" signal, which begs the question others have asked... how can these people choose to act in the way you want and yet not have been discovered long before they could, themselves, detect a potential threat and act to avert it?
Answer: They wouldn't.
Humanity is vastly more advanced than them, and their planet is suitable for life. They have no real way of beating 1000 years plus super tech from earth.
They need to start their own colonization program.
160 light years is a long way. They can work on expanding out through their own solar system, colonizing all major resources, and fortifying it. Even a much more advanced earth is gonna have trouble conquering a whole solar system without ftl tech. Grab as many systems as you can, devote a huge amount of the budget to space travel and space techs, and build an enclave which can resist humanity or flee.
Answer: One way they could avoid detection is if the aliens and their entire life forms are sufficiently different from life on earth. Namely they life deep inside a gas giant. There system doesn't contain any planets that would be interesting colonizing targets for humans so no one goes there. Gas giants are plenty in space so there is no reason to go to that stellar system.


Determining distance between wireless sensors - sailing project

Question: I'm looking for a solution which will allow me to determine the distance between two points, one of which is static and the other not. It's part of a project I'm working on which will involve the creation of a sailing app for my boat. Based on a wireless network of sensors, I want to be able to determine and display on a tablet, information which shows how far the sails have been hoisted up on the mast. I need to put some sensors on the mast and in the sails themselves. I worked on RFID tags but got poor results due to high power consumption. I tried ZigBee distance measurements and accelerometers but ended up with a lot of noisy data. Any ideas on the matter would be much appreciated. Please remember that there are environmental matters to consider (weather) and the fact that the boat is in constant movement.
Comment: How about mesuring the length of the rope used to hoist the sail? Could be done either directly by trying to find out how much rope went up/down the mast, or indirectly by figuring out how much of the rope is still on the winch, for example.
Comment: How high is your mast (distance to measure)? What is the precision needed?
Comment: The mast varies in length but let's say 30 m. The precision needed is approx. 1 m. We've thought about the rope but it gets a bit complicated. We need to feed data into the app so getting it from the sails instead of the rope seems a little easier.
Comment: You could use a magnet and hall sensor on the masthead turning blocks to count revolutions. Need to be able to detect direction of revolution also. Anything on the sail is going to be difficult to power up. Also, sails flog around when you luff or tack and get caught in the rigging and so on.
Answer: Just brainstorming here.
You could put barometric pressure sensors on the head and tack of the sails. If you measure the pressure difference you can calculate height. Might not be super accurate, but it would probably work to some extent. The short term change in pressure in the head sensor would be somewhat accurate while the sail is going up, even without reference to another sensor.
You could also put one sensor on the base of the mast to be your zero reference, and in that case, each sail would have just one sensor at the head. This would work better for conventional spinnakers which have considerable variation in tack height, depending on conditions.
Another way is to use something like an optical encoder on the mast. The detector would be on the head of the mainsail (this would only work for the main, probably not for headsails).
None of these things are going to be super reliable. In other words, I don't think this holds promise as an industrial type of system. It would be more like a teaching aid that could be kind of interesting (basically a gimmick). If you need rock-solid reliability, I think you will need to consider special halyards or something like that.
Comment: Great idea.  Two barometric sensors, protected from the wind.  http://www.st.com/web/en/catalog/sense_power/FM89/SC1316/PF261381  should do the job nicely
Comment: I think the optical encoder equivalent is a good one. Sails (or at least the head of the sail) ride on a track through the mast, so you could actually have some reliability there.
Answer: I kinda like the idea of Ultrasonic measurement here.  There would be two transmitters: one is battery-powered and mounted on the sail.  It would emit a chip periodically.  The other transmitter is with one of the receivers.
You would have TWO receivers: one at the top of the mast, the other at the bottom of the mast.  If I am thinking correctly, the ratio between the received times would give you the position of the sail on the mast.
Note that you do NOT need to know exactly when the transmit chirp occurred.  That is: you don't need any connection (wired or wireless) between the transmitter on the sail and your electronics.  Instead, you simply see which signal arrived first: the base or the top of the mast.
You calibrate the system by firing the reference transmitter (located with one of the receivers) and measuring the time for that signal to be received by the other receiver.
You could also encode a signal into the sail transmit chirp if you have multiple sails.  This allows you to ignore signals from a sail that is not on that particular mast.
Comment: Wouldn't the constant changing wind speeds around the boat severely affect the accuracy of any kind of ultrasonic measurement? Strong winds may even just blow the signal into oblivion. I would go for something passive with hall sensors or the like. The rope seems like the best place to start.
Comment: Wind is primarily horizontal. Speed of sound is much faster than wind, so horizontal wind shouldn't affect the signal too much. Narrow-band detection will help defeat wind noise. If you want to get really fancy, you can use a complex signal with correlation detection. Processing gain.
Comment: @captch: That is a possibility.  But I think that the idea is worth exploring.  It's fairly trivial to make ultrasonic transducers weather-tight (think Fish Finder) and almost anything else that has been suggested just isn't sufficiently robust.
Answer: You could try it with a ultrasonic time measurement. You place a ultra sonic speaker on one side of the measurement and a receiver on the other. By measuring the time of flight you can get the distance in between.
Comment: If you do this, you need to control for air temperature. One way to do that would be to have two sensors at a known distance. Measure the time of flight over the known distance to calibrate.
Comment: Oh, I better add, the speed of sound is controlled by air temperature almost exclusively. Don't worry about air pressure. It might seem like that should be an important factor, but it is not. And in any event. this calibration method will control for all variables.
Comment: The idea is to keep the system simple and reliable. The boat moves constantly in very different and difficult weather conditions. Calibration may be a problem as you may find yourself in weather conditions which do not allow it for even a few days. I think the system has to be as weather proof as possible.
Comment: Sorry for not being clear. The calibration should be automatic and more or less continuous. There could even be a dedicated transmitter and receiver strictly for calibration. I don't think I would worry about air temp differences at different heights. It could happen but will generally not be the case. None of the proposals so far strike me as simple and reliable in the real world. Ultrasonic tends to suffer from "stuff" clogging up the grill. Could be seawater, dust, etc. All shipboard electronics are subject to corrosion. It is an interesting but challenging problem.
Answer: If discrete measurements are OK you could install magnets on the mast and a hall effect sensor on the sail (or vice versa).
Comment: If it has to work for headsails, this probably won't work very well because they don't ride up the mast on a track. It could work well for the main sail, however. It would be best to have one magnet on the top of the main, and multiple detectors on the mast.
Comment: This seems pretty interesting although I don't really know how reliable this method would be however I'll have a look into it. Ideally I'd need a system that works on all sails and not just the main sail ...


How can I decide which cricket bat to purchase?

Question: I'm 15 years old and just beginning to play cricket. While I was researching bats I decided to get a light weight bat (around 2.6 lb) - but how do I find out what size I need or what company bat I need?
Answer: If you have a sports shop near you that stocks cricket kit, go there and examine all the bats. Lift them, handle them, see how they feel in your grip. Find one or more than feels right, picks up well, has your preferred weight etc.
When you find one or more that you like, examine the prices. See if you can afford your favourite, or see if you can get the same bat type online within your budget.
I would encourage you to buy from the local shop though, if you can. That keeps them in business.
Comment: There are no cricket stores near me.
Comment: That's a pity. You then need to pick a bat online - so long as you get the weight you say you like, and the correct size for you (there are plenty of sites to advise you on this) then it doesn't really matter what company made it.


Сhanging the camera path

Question: I need to change the camera path at key points. When moving the curve vertices after empty, the camera stays exactly where I want it. But when editing vertices in front of empty, the camera changes its position
Answer: Maybe try this solution: Create a cube (or a triangle), select the camera, shift select the cube, switch to Edit mode, select 3 vertices, press CtrlP > Make Vertex Parent:

Give your cube a Curve modifier with the curve as Curve Object:

Now when you increase or reduce the length of the curve the camera doesn't move:
Comment: Très bien! but... if I change the position of the first vertex of the curve, then the camera will change its position
Comment: oops yes you're right

Сhanging the camera path


10.04 LTS SVN how quick are updates?

Question: I am new to Ubuntu LTS.  I am using the SVN server that comes with 10.04 it is 1.6.6.  The current version is 1.6.17. How quickly can I expect an update to the LTS applications?
Answer: Generally the LTS will not get new versions of applications, but rather security patches and bugfixes to current package version - the version number will likely never change, however there could be new features backported, but not likely. Your best bet would be the Subversion PPA but that only goes to 1.6.17.

What are PPAs and how do I use them?

If you are wanting something newer than than, you will likely have to install from source unfortunately.
Answer: Important security bugfix updates you can expect very quickly, but it will be on base of 1.6.6 (base vesrion for this release) with merged only security changes, so you will have package version like: 1.6.6dfsg-2ubuntu1.3. See changelog.
So, if you care about security, stick with normal package from normal repo.
If you care about some new features, you need to:

compile it by your self (and take care about updates by your self!), or
install some PPA package without support, or
install package from newer ubuntu release

All of above are messy - do you really want it?

10.04 LTS SVN how quick are updates?


scaling magnitude of the dft

Question: I have seen a lot of question like this, but still I have doubts about the answer.
When using the fft in matlab the solution need to be divided by the length of the signal (N).
I have seen many answers that say that there are at least 3 type of normalisation:
1. 1/N  (I have seen to use this a lot of time)
2. 1
3. 1/sqrt(N)
What is difficult to understand to me is when I should use each one of this 3 normalisation. It is difficult to me to understand why this 3 different normalisation can be exact together. If I want to obtain the right amplitude which one should be chosen and Why?
Another thing, I was thinking the fact that if we look at the fourier coefficient, there is a normaliation 1/N. Is this the explanation of using this normalisation?
Answer: $1/N$ is the correct scaling to have the resulting DFT output represent the average for the input signal that is rotating (frequency) at that particular bin in the DFT.
This is very clear when considering bin 0:
$$F[k=0] = \frac{1}{N}\sum_{n=0}^{N-1}x[n]e^{-j0} = \frac{1}{N}\sum_{n=0}^{N-1}x[n]$$
Where we see it is simply the average. For signal at any other frequency, the DFT essentially de-rotates that signal to 0 and then takes the average. It may help to realize that the form $e^{j\omega t}$ represents a complex spinning phasor with magnitude 1 and frequency of rotation $\omega$, spinning counter-clockwise in time. Similarly the form $e^{-j\omega t}$ is a complex phasor spinning clockwise in time.
So if you considered another example signal that was completely at bin 1 once the DFT is taken, this would the spinning phasor $e^{j\omega_0 n}$ in time where $\omega_o$ rotates ones as we count from $0$ to $N-1$ and n is the discrete sample index in time. So specifically $x[n]=e^{j2\pi n/N}$.  Look what we do under the hood in the DFT equation with $x[n]=e^{j2\omega_o n}$:
$$F[k=1] = \frac{1}{N}\sum_{n=0}^{N-1}x[n]e^{-j\omega_o n}$$
$e^{j2\omega_o n}e^{-j\omega_o n} = 1$ so we again arrive at the moving average for the signal that was at that particular frequency.
To answer @relayman357 challenge with this the correct scaling of $1/N$ does makes sense as given by Euler's identity, we see that the average value for each of the two complex tones in a real sinusoid is 1/2!
$$cos(\omega t) = \frac{e^{j\omega t} + e^{-j\omega t}}{2} $$
So if you want the result to accurately represent an average quantity, you divide by N. However this scaling is arbitrary and $1/\sqrt{N}$ is also commonly used as it makes the DFT and IDFT completely symmetric since that same scaling can then be used in either direction.
Answer: Luca - this is a good question.  If you use the approach here the magnitudes of the resulting harmonics are wrong.  I can't figure that out myself.  For example, if you take a sine wave with amplitude A and run it through, the resulting Xk will be wrong by N/2.  So, you have to scale them by 2/N to get the original amplitude A.
Power engineers like quantities in [RMS] so you would add a square root of 2 in the denominator (2/N/root2) to get RMS.
I'd love the math folks to help us with this - maybe they don't care about the actual amplitude of each harmonic?  Maybe all they care about is the angle or the relative magnitudes...
Comment: The reason it appears "wrong" is that the Fourier Transform is telling you the magnitude of each exponential not each sinusoid: From Euler's identity you get the relationship between the two: cos(wt) = 1/2 e^(j wt) + 1/2 e^(-j wt)
Comment: Well, there we have it.  Thank you @DanBoschen - much appreciated.
Comment: Glad you see it with that super short explanation. When you do a complex conjugate product of two functions and then integrate the result - that is a "correlation" and it will maximize in signal to noise ratio (for uniform white noise) when the functions are equal - as the phases in each product will cancel and you accumulate a big sum all aligned. This is what is happening in the FT: we correlate to each e^(j 2pi f t) term and the result shows us the amount of each of those terms in the original waveform —- thus a sinusoid ends up with two peaks since it has two such terms.

scaling magnitude of the dft


Centrifugal Force vs Motion (inertia)

Question: Could centrifugal force be appropriately described as motion (inertia)--rather than an (imaginary) force?  Could such a change of perspective eliminate much of the common confusion?
Comment: This is correct and is actually the consensus in Newtonian Mechanics: Centrifugal Force is the the name given to the spontaneous acceleration experimented by bodies as measured within a specific type of non-inertial reference frames. Those frames being rotating when observed from some inertial reference frame.
Comment: There is nothing "imaginary" about centripetal forces. Ultracentrifuges with up to 2000000g of acceleration are commercially available and the mechanical construction of these machines requires enormous care for the real forces that are being generated by their very fast spinning innards. The confusion you mention arises about fictitious force. i.e. the seemingly weird movement of inertially moving bodies when seen from non-inertial coordinate systems. Motion of bodies in a circle requires a real force, not a fictitious one.
Answer: It is indeed like this. Centrifugal force is the virtual force experienced in a non inertial frame that is rotating, the reason why we always hear about it and rarely about its "real" counterpart, centripetal force, is because we tend to set our mind in the simplest referential frame to interpret. So if we are in a car or in a bus and it turns, ore even if we are just looking at it from the outside, we will instinctively put ourselves in the bus frame, where thing will accelerate toward the direction opposed the center of the curve so if we want to apply Newton laws in our referential frame we have to suppose a force directed that way, centrifugal force indeed. If we analyze the same thing from an inertial frame than we see that things are just going straight and the bus is accelerating in the other direction.
Answer: This is a big source of confusion, and from a pedagogical standpoint, I'm a fan of trying to eliminate the terms "centripetal force" and "centrifugal force" wherever possible. In the classroom, I try to emphasize that "centripetal" and "centrifugal" are just directions (towards and away from the center of a circle respectively) just like "up," "down," "East," and "Into The Page."
We don't talk about an "up force" or an "east force," -- we say "the ball feels a net force in the upward direction." Similarly, I try to get people to think about what's behind the centripetal or centrifugal forces. There's always a reason for the force (gravity, rocket engines, etc.) and if not, it becomes clear that the force doesn't really exist.
Here's an example of what I'd want a student to be able to say: As a car goes around a curve, there is a friction force by the car seat on the passenger's back, and rear end pointed in the centripetal direction. This force keeps the passenger accelerating into the curve. Meanwhile, the passenger feels as though something is pushing her in the centrifugal direction. On closer examination though, she realizes that there is no force pushing her outward, so she realizes that this is just a consequence of Newton's First Law, which says that her body would prefer to continue in a straight line.
If I ruled the world, I would get rid of the words centrifugal and centripetal and fictitious force because they are clumsy, and easily described by more common words/concepts. Of course I don't, so I still talk about them!

Centrifugal Force vs Motion (inertia)


What's the difference between an ecological study and a cross-sectional study?

Question: What's the difference between an ecological study and a cross-sectional study? And could you provide an example?
Answer: An ecological study is one where you take values of a variable for an entire population, the outcome from that population, and use that to draw inference. A cross-sectional study is where you look at individuals within a population at a single point in time.
Examples:
Ecological Study: Consider a study where you find the average number of years of education each of the 50 states in the U.S., as well as their rate of teenage pregnancy, and drew a correlation between them.
Cross-Sectional Study: You recruit people in Montana, and ask them about a number of biological, sexual and behavioral variables and their HPV status, and examine whether particular variables are associated with having (or not having) HPV.
Answer: DEFINITIONS
An ecologic study is a study that uses at least one measurement made at the group level.
A cross-sectional study uses measurements all of which are made on individuals.
https://www.ctspedia.org/do/view/CTSpedia/StudyUnit
ECOLOGIC STUDY EXAMPLES
Many good examples of ecologic studies are studies that assess disease rates according to population measures of nutrient consumption for countries across a wide range of consumption levels.
EXAMPLE 1
An often-cited and classical study is Armstrong and Doll's comprehensive assessment of various dietary factors and cancer mortality and incidence by country.  In the study, information about the incidence and mortality for 27 cancer sites and a large number of population measures of diet was obtained from public sources for 37 countries.  A graphical depiction of the results of the analysis of colon cancer incidence in women according to per capital daily mean consumption in the 23 countries with data on colon cancer incidence and meat consumption in women is shown.
Armstrong B, Doll R. Environmental factors and cancer incidence and mortality in different countries, with special reference to dietary practices. Int J Cancer. 1975 Apr 15;15(4):617-31. PubMed PMID: 1140864.  FIGURE 2.  Correlation between incidence of colon cancer in women and per caput daily meat consumption in 23 countries.

https://onlinelibrary.wiley.com/doi/abs/10.1002/ijc.2910150411
EXAMPLE 2
Another ecologic study of cancer and diet, with an excellent graphical presentation of the results (FIGURE 1), is a study that assessed the relationship between stomach cancer mortality and dietary salt and nitrate in 24 countries.  In this study, population-level data from 24-hour collections of urine that measured dietary salt and nitrate were used in conjunction with data on stomach cancer mortality for 24 countries to assess the relationship between stomach cancer mortality and dietary consumption of salt and nitrate.
Joossens JV, Hill MJ, Elliott P, Stamler R, Lesaffre E, Dyer A, Nichols R, Kesteloot H. Dietary salt, nitrate and stomach cancer mortality in 24 countries. European Cancer Prevention (ECP) and the INTERSALT Cooperative Research Group. Int J Epidemiol. 1996 Jun;25(3):494-504. doi: 10.1093/ije/25.3.494. PMID: 8671549.
https://academic.oup.com/ije/article/25/3/494/756987
CROSS-SECTIONAL STUDY EXAMPLE
This is a cross-sectional study of the association of dietary intake of calcium, sodium, and potassium with blood pressure.
Morikawa Y, Nakagawa H, Okayama A, Mikawa K, Sakata K, Miura K, Ishizaki M, Yoshita K, Naruse Y, Kagamimori S, Hashimoto T, Ueshima H. A cross-sectional study on association of calcium intake with blood pressure in Japanese population. J Hum Hypertens. 2002 Feb;16(2):105-10. PubMed PMID: 11850767.
https://www.nature.com/articles/1001314
In the study, 476 men and women age 20-59 years were included.  Blood pressure was measured and dietary calcium intake was estimated from a 1-day 24-hour recall in the 476 individual participants.  Dietary intake of sodium and potassium was measured using 24-hour urine collection in the 476 individual participants.  Individual-level data were analyzed to examine the association of dietary calcium intake with systolic blood pressure after adjustment for dietary potassium and sodium separately in men and women.


Excel - Create text file named after a cell containing other cell data

Question: I'm looking to create a .txt file for each value in column A containing the corresponding values in columns B and C
Answer: The more I looked at this the more I found it to be a useful little macro. In order to keep it from processing blank rows (and locking up my Excel) I rewrote the code to only create a file while there is data available. Also, the use of print rather than write creates text without the quotations. Here is what I used to accomplish the same thing.
Sub CreateFile()
Do While Not IsEmpty(ActiveCell.Offset(0, 1))
    MyFile = ActiveCell.Value & ".txt"
    'set and open file for output
    fnum = FreeFile()
    Open MyFile For Output As fnum
    'use Print when you want the string without quotation marks
    Print #fnum, ActiveCell.Offset(0, 1) & " " & ActiveCell.Offset(0, 2)
Close #fnum
ActiveCell.Offset(1, 0).Select
Loop
End Sub

Feel free to use and modify as you wish. Thanks for the great idea.
Answer: This macro will take each value in column A, produce a .txt document in the name of the value and insert the corresponding information from columns B and C
Sub CreateTxt()
'
For Each ce In Range("A1:A" & Cells(Rows.Count, 1).End(xlDown).Row)
    Open ce & ".txt" For Output As #1
    Write #1, ce.Offset(0, 1) & " " & ce.Offset(0, 2)
    Close #1
Next ce
End Sub

Four years later I decided to come back to my first SU question - my very beginnings to VBA. This is better, but will overwrite any file that already exists
Option Explicit

Sub CreateFileEachLine()

    Dim myPathTo As String
    myPathTo = "C:\Users\path\to\"
    Dim myFileSystemObject As Object
    Set myFileSystemObject = CreateObject("Scripting.FileSystemObject")
    Dim fileOut As Object
    Dim myFileName As String

    Dim lastRow As Long
    lastRow = Cells(Rows.Count, 1).End(xlUp).Row
    Dim i As Long

        For i = 1 To lastRow
            If Not IsEmpty(Cells(i, 1)) Then
                myFileName = myPathTo & Cells(i, 1) & ".txt"
                Set fileOut = myFileSystemObject.CreateTextFile(myFileName)
                fileOut.write Cells(i, 2) & "    " & Cells(i, 3)
                fileOut.Close
            End If
        Next

    Set myFileSystemObject = Nothing
    Set fileOut = Nothing
End Sub

Comment: Great idea. This code created the text files for the data I have in two rows, but then causes Excel to stop responding. The line `For Each ce In Range("A1:A" & Cells(Rows.Count, 1).End(xlDown).Row)
` returns over a millions rows even though only two are populated.
Comment: That hasn't happened to me, but I think I select it and use a keyboard shortcut to run the macro. Ideally, I should put in my range, but recently it's been varying too much
Comment: Additionally, you could have it start at the bottom and `xlUp` to the top to avoid there being blank sheets. I'm wide open to a more elegant solution.


Create a Pop-Up menu to call a custom exporter (Blender 2.6 API)?

Question: am trying to create a friendlier menu to export models using a file format I created, using the python scripts in Blender 2.63.
Have been checking the API documentation for Blender, also the template examples and haven't figured out how to actually call the Exporter I created, which works ok.
Let me describe a bit more the problem; to export I have to go to File->Export->select the desire format->give a name->Press Export button. I want to automate this process through a menu (like the one provived in the templates, ui_menu); just load/run the script and select Export, the rest should be handled by the script it self. Lets suppose the blend file is MyCoolModel.blend, it will assume the exported file will be placed in the same location, taking the name MyCoolMode appending the correct extension.
By having the Custom menu, which is in one file. Don't know how to call the proccess of the exporter giving the described parameters, don't even know if that's possible.
On the other hand, it might be easier to move the code of the exporter to another module, the Custom Menu, add a button and call a function executing what was described, I just one to get feedback if someone already coded this, and re-use the export module.
In the end I would want to preserve the original export module, in case the artist needs to change the default name of the exporting process, or simply 'cause he wants to do it manually.... Having both versions, trying to re-use the Export would be the best if that's possible, no duplication of code really ;).
Answer: I am looking to do something similar, they only thing that I have found that is close enough is the following Scripts/Cookbook/Interface:popup. Of course is still far away from anything close to the export menu, but hopefully it can be extended a lot more.


In 2 player, 10 card Gin, figuring out opponents cards

Question: What tactics can you use to help determine what your opponent is looking for in two player 10 card Gin?
Answer: In Gin, you don't get any hard information on what is in your opponent's hand until the hand is over.
This means you have to make use of whatever information you can get.  The primary source of information is the discards.  To be successful you need to keep track of both

What your opponent discarded
What you discarded

Pay attention to both suits and the number of each card.  For example, if your opponent discarded the 5 of clubs, he is most likely not interested in 5's or mid range clubs.
Observe your opponent, if he usually plays fast but pauses even briefly to consider the jack of diamonds you just tossed then he most likely has several cards around that.  A couple examples of what he might have.

10-K of diamonds
sets of 9's and 10's but he has diamonds of both and could make another run (like 7-8-9-10 of clubs)

Perhaps from previous discards and pickups you could narrow down the options as to what he might have.
Comment: I like your answer but I think that the first statement: "In Gin, you don't get any hard information..." is wrong.
Answer: In Gin, you do get some hard information on what is in your opponent's hand and constantly need to add some intelligent guesses on what is probably the opponent's hand.
Hard information:

A pickup from the discard by the opponent - surely that opponent hold this card. Probably this card is a part of a Meld or at least a combination.
Your hand & discard pile - surely opponent do not hold those cards.

Softer information:

Opponent's discards - almost surely are not part of a meld, that is to say, opponent probably does not holds two cards in the same suit with rank {+1,+2,-1,-2} or two cards in the same rank.
Cards that cannot make a meld since other cards are discarded.  For example, the K♠ is probably not in the opponent's hand if the Q♠ and two other Kings were discarded already.

The nice book: How to win at Gin-Rummy by Pramod Shankar suggests to keep a table of the opponent's likely hand and to mark any card that is for sure in or for sure not in the opponent's hand, for example:


Transposition for baritone saxophone in EFlat

Question: Is this the correct transposition for a baritone saxophone in eflat pleast? What will the new key be?
Comment: Take a look at a previous question - Modulation theory exercise for baritone sax (Eb).
Answer: When a tranposing instrument pitched in E flat plays a written C, the actual note produced is E flat.  So, a minor third higher or a major sixth lower or an octave or more away from that.  Knowing that the baritone sax is pitched in E flat alone does not tell you which octave.  You need to know.  The alto sax is a major 6th lower and the baritone is an octave and a major 6th lower.  So, you need to compensate for this shift by writing an octave and a major 6th higher.
Your music is in F.  A major 6th higher is D so you should write the sax part in D.  Your first note is F and you have correctly moved it up an octave and a 6th.
Actually, the two are linked.  Since it opens on the tonic, figuring that the sax part opens with D tells you that the key is D.
Another way to get the key is to know that the sax will have 3 more sharps.  Think of flats as negative sharps.  Concert pitch is 1 flat so -1 sharp and the sax will have 2 sharps.


If you are falling, can you Misty Step to the ground to avoid damage?

Question: If I fall off a cliff - say, 1000 feet - can I wait (readied action?) until I am near enough to the ground (my understanding is you fall 500 ft/round) then misty step safely to the ground? Or, does the momentum from the fall carry over through the teleport and I still splatter across the rocks?
Comment: Related: [Do teleportation spells conserve momentum?](https://rpg.stackexchange.com/questions/108686/do-teleportation-spells-conserve-momentum), [What happens if I Blink while in free fall?](https://rpg.stackexchange.com/questions/124334/what-happens-if-i-blink-while-in-free-fall)
Answer: You can't ready bonus actions. They can only be done on your turn. and so you can't ready misty step as a bonus action (you need a bonus action to use misty step) and use it to survive the fall.

A bonus action can be taken only on your turn. You, therefore, can't
  use Ready with a bonus action. #DnD

The lead rules designer confirmed it.
As V2Blast noted, the top answer to this question says that momentum is conserved.
That said, you could ask your DM about this.

When you describe an action not detailed elsewhere in the rules, the DM tells you whether that action is possible and what kind of roll
  you need to make, if any, to determine success or failure. p193 PHB

I might allow an acrobatics skill check to use the spell to alter your momentum, and land somewhere safer. Portal style momentum alteration, not negation.
Comment: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/110473/discussion-on-answer-by-nepene-nep-if-you-are-falling-can-you-misty-step-to-the).
Comment: Using the abstraction of the turn structure and action types in this situation seems quite strange. Maybe mark it explicitly as a strict and unintuitive rules as written solution, which only applies if we are in the framework of combat initiative for the entire falling time.
Answer: Yes, you can avoid fall damage using Misty Step by casting it on the round you would have crashed into the ground.
If you fall for 1 round and within the next round successfully cast Misty Step before hitting the ground, you would be able to avoid taking fall damage according to the top answers of the related questions V2Blast linked.
Potential problem.
You wouldn't be able to instantly cast Misty Step right away because the farthest you can teleport is 30 feet and you would still be in the midst of falling the last 500 feet. Since bonus actions can't be Readied, but are only cast on your turn, you can't cast Misty Step as a reaction.
According to the following tweet, a bonus action cannot be readied to be taken on a different turn. However, if you are the only one falling and your turn lasts for the duration of the round, you could reasonably wait until the end of the round to cast Misty Step to avoid the fall damage (emphasis in tweet mine).

A bonus action can be taken only on your turn. You, therefore, can't use Ready with a bonus action. #DnD

Ruling for a DM.
Your DM could have you make an intelligence or wisdom check to see if you judge correctly how far away the ground is when you cast the spell, but that's merely a suggestion.
Answer: Outside of combat, bonus actions can be taken at any time, so it is possible to cast misty step in this way.
Whether or not you have the reflexes to cast misty step when you are less than 30ft from the ground is another question. In my games, I call for a dexterity saving throw for anything reflex related.
Whether or not teleportation spells conserve momentum is up to your group to decide together. Don't wait until you are already falling to have this discussion.
Comment: Good point about the timing vs altitude! I do think this answer begs the question about whether or not the character is (or should be) in initiative, though. If things are happening quickly, it seems like initiative *ought* to be used. But even if it's not, the first answer you link to doesn't quite say what you're using it to say. That answer says combat status has no bearing on a character's ability to use bonus actions. It doesn't say a player has carte blanche to ignore timing considerations.
Comment: @Rykara Yes, that's why I suggest using a dex save for the timings. Remember that initiative is (RAW) for combat. It's not just for anything where timing matters. This case is a great example of why, if you use initiative then you make it impossible, that's clearly not a good idea. I chose to address out of combat because the current upvoted answer addresses ONLY in combat, leaving some ground uncovered.


Is there a noncomputable set which can be described by a probabilistic Turing machine with bounded error?

Question: Does there exist any noncomputable set $A$ and probabilistic Turing machine $M$ such that $\forall n\in A$ $M(n)$ halts and outputs $1$ with probability at least $2/3$, and $\forall n\in\mathbb{N}\setminus A$ $M(n)$ halts and outputs $0$ with probability at least $2/3$? What if you only require that $M(n)$ is correct with probability greater than $1/2$?
Answer: Every such decision problem is computable, even in the harder version of the problem, assuming that the transition probabilities are, say, fixed rational numbers.   A deterministic algorithm can calculate the probability distribution on the set of states of this stochastic TM after each $t$ time steps, and then step through $t$ until the probability of halt at either yes or no exceeds $1/2$.
Comment: Every problem in BPP is indeed in the polynomial hierarchy, and therefore in the computable hierarchy, but that is not what I'm saying.  Algorithms in BPP are a priori time bounded, which can be interpreted as a second reason that everything BPP is computable.   By contrast a general recursive algorithm is not a priori time bounded; it only finds its own reasons to eventually stop.   Thus, the result that probabilistic-computable implies deterministic-computable does not reduce to a question about BPP.
Comment: In other words are you saying $BPP$ is in computable hierarchy (since it is in polynomial hierarchy)?
Comment: @AJ. Yes, every problem in BPP is computable.
Comment: @GregKuperberg thank you for the clarification. would irrational probabilities make a difference?
Comment: For the question as stated, they would only make a difference for an unfair reason.   If a transition probability is itself an uncomputable real number, then its bit expansion can be extracted by a Turing machine.   On the other hand, if there are finitely many transition probabilities and they are all computable, then a slightly more complicated argument shows that everything that the TM can do is still computable in the usual sense.
Answer: Construct $A$ as follows.  Roll a 6-sided die infinitely many times, giving output $r_1,r_2\dots$.
Now for odd $k$, say $k\in A$ iff $r_k=6$.  For even $k$, say $k\in A$ iff $r_k<6$.  So $k\in A$ with probability 1/6 if $k$ is odd, and $5/6$ if $k$ is even.  $A$ is obviously incomputable.
Turing machine $M$ on input $k$ simply answers 1 if $k$ is even, $0$ otherwise.  So it is correct 5/6 of the time.
Comment: That doesn't work. Let $k$ be even with $k\notin A$. Then $M$ will guess that $k\in A$ and be wrong with probability 1. I wanted $M$ to be correct with probability greater than $1/2$ on every fixed $k$, not just for it to be right about most $k$.


Does Internet speed depends on processor?

Question: I'm about to purchase tablet computer. My main purpose will be to browse Internet. So, please tell whether the processor speed has to do anything with the Internet speed. Different tablets have different processor speed like 1GH, 1.2GH, 1.3GH, 1.6GH etc.
Comment: The CPU clock alone (assuming same number of cores) is not enough to compare different tablets: you have too look at the whole picture, and consider the amount of available RAM, internal storage, the graphic card, operating system, and so on.
Answer: Internet's speed is the rate at which you can transfer data over the Internet. As Jonas Wielicki wrote in his answer, this should not be affected by CPU speed, at least not in a noticeable way.
User experience will be affected, though, because when browsing World Wide Web (which are not the same as the Internet - the former means websites, the latter is a global network used to transmit those, but also to send any other data - file transfers, Skype calls etc.) actual display times are more important than download speed.
Fast Internet connections (either over Wi-Fi or cellular connections) are widely available today and download speeds for most sites aren't that important. Here's a screenshot from Chrome's profiler showing load time breakdown for SuperUser on my current hardware and connection:

As you can see, loading takes only 67 milliseconds, while rendering, painting, running scripts and mysterious other tasks add up to one second. Downloading makes only a fraction of entire load time, as clearly visible on the pie chart. Other steps of displaying a website heavily depend on hardware, so CPU will matter when browsing.
So hardware is important if you're going to use that device for browsing WWW. There is no mobile CPU that will be definitely "fast enough". The higher the frequency is, the better it will perform. Number of cores also matters. Other factors may limit the CPU, like amount of RAM, its speed, storage memory's speed, operating system version and modifications applied by manufacturer.
Answer: Not for speeds you currently get via wireless, if you don't end up having a extraordinary bad wireless chipset.
Machines with about 1 GHz may have problems with gigabit speeds and bad ethernet chipsets (I have heard such reports from people with homeserver systems), but wireless is still some factors away from that.
Mobile internet is even farther away from gigabit speeds, so no issues should occur here.
Comment: A higher CPU speed will process Javascript better, which may make browser pages more responsive, so it may make your Internet "seem" faster (even though it hasn't actually changed) from the perception of many people.
Comment: This, @ultrasawblade, is a very good point I have not thought about at all. It is a bit unclear what the OP is asking, although the internet-connection tag suggests its indeed about transfer speed. In case it is not, your comment is certainly worth an answer.
Answer: If you are running TCP speed tests it has a lot to do on your CPU.
1: Open up task manager, have a look at your CPU
2: Open up Chrome, have a look at your CPU.
3: Run a speed test and see the CPU rise.
If you have an old CPU and its maxed out at 100% the test would not be able to show you the full speed you could receive.
Run on a new CPU and it will show better numbers.
Again this is just when you run speed tests, the internet would always be the same speed as you receive it from the ISP, it just means some computers can not handle the JAVA, Flash downloads.
Answer: CPU makes a huge difference, unless it's my RAM...not sure, but the difference between these 2 machines is absolutely phenomenal, I'm talking up to 4 times fast
Laptop:
with 6 Gigs Ram
Intel B960 CPU
VS.
Desktop:
16 GiGS ram
i5-8400 CPU
Mega difference, now is it the RAM or CPU? Not sure, but it doesn't take ram to open up single pages I think, and I did tests with both machines idling at under 2 gigs or RAM
Answer: I have tested the CPU effect on Internet speed myself. I tested a single core desktop, duo core laptop, quadcore cellphone, and quadcore tablet, same specs on the phone and tablet.
I get 17.1 mb download on the phone and tablet, but only 6mb download on the single core computer. The tablet has even reached 34mb download, when it is in the same room as the wifi router.
The desktop single core had 4 gigs of RAM. The tablet and phone quadcore have only slightly more than 1 gig of RAM, and they are both almost maxed out.  So CPU is the only major factor.
As for the duo core laptop, it has similar download speed to the desktop, but maybe about 5 mbs faster.
CPU matters.
Comment: You're answer attracted a downvote because it was more rant than answer.  As a serving suggestion, we've removed the extraneous commentary to focus on the meat of the answer.  You can always roll it back if you disagree, but the present form is an improved answer, and is consistent with the site's focus on the technology aspect.
Answer: My desktop, laptops AND Smartphones can utilize 98 Mb/sec of my cable's 100 Mb/sec throughput modem, but tablets, and Chromebook (Android ~ but so are Smartphones ?) only obtain 50 Mb/sec download.  This is true for 2.4 MHz or 5.0 MHz wi-fi......  Smartphones and Laptops are as fast as hardwired Ethernet on desktop!
Answer: It will definitely affect the download and upload speed as the processor will be required to crunch all that data and process it to be shown to u accordingly using various softwares like Java etc. even the latency affects your PC performance try uploading and downloading while using a data crunching software. And u will get different speeds using different devices at the same location. So even a minor change in the configuration will be reflected. Try testing it on two different devices at the same time and u will see a the difference
Comment: I am sorry, there is absolutely no truth in this answer, there is no direct connection between your internet speeds and your CPU
Comment: If the CPU is the bottleneck, Internet speed would be the least of your problems.


What are SIDS and STARs?

Question: I've been exploring the FMC(flight management computer) on the PMDG 777 and noticed that under the route section you can insert VIA and TO points. From what I've learnt this is where you would insert SIDs and STARs. What are SIDs and STARs? And how are they different?
Comment: Take a look at https://www.faa.gov/air_traffic/publications/media/AIM_Basic_dtd_10-12-17.pdf, in particular 5−2−8 and  5−4−1
Comment: The "S" means standard. So you need to know what is a departure, and what is an arrival first. Being standard helps, because everyone knows about the path and constraints associated with it. So ATC and pilot can talk about it just using the name, no need to discuss about the waypoints or altitudes that define the departure procedure or the arrival procedure.
Comment: [This site](https://www.vatsim.net/pilot-resource-centre/ifr-specific-lessons/sids-and-stars) has good information explaining the concept.
Answer: Both are predefined routes through the air which connect the airport areas (terminal areas) to the enroute airway system in a way remotely similar to local roads that everyone follows from and to highways. This ensures (in theory) an orderly fashion of after departure reaching (SID) and leaving towards the airport (STAR) the enroute airspaces for all aircraft.
SID stands for Standard Instrument Departure.
STAR stands for Standard Terminal Arrival Route.
The links touch upon everything else.
Comment: An additional important feature of SIDs and STARs is that they simplify ATC clearances and flight plan programming.
Answer: The "VIA" and "TO" on the PMDG 777 can accept Airways, SIDS and STARS and their transitions.
SID Stand for Standard Instrument Departure, which is a predefined route involving GPS waypoints or navigation aids. It prevents chaos near the airport airspace by having everyone exit the airspace the same way.
STAR Stands for Standard Terminal Arrival Route. This once again is a predefined route to enter the airspace Ina orderly fashion to land on a runway.

What are SIDS and STARs?


How to prove that shadows in the cave aren't talking?

Question: If a physicist (who have never seen people) is constantly watching on shadows and hear voices behind, will he make a statement that shadows are talking? I'm talking about physical experiment in terms of Allegory of the Cave.
Comment: Does this physicist know about ears and how they work? It is difficult to turn this thought experiment into a scenario that makes sense *and* is detailed enough that one can reason about it fruitfully.
Answer: Assuming that you are talking about a scientist standing alone in a cave talking and trying to see if his shadow is the one talking or not:
He can just stand where the light will project the shadow far enough to see the discrepancy. However, to arrive at the conclusion, he might need some kind of echo to be present in that cave so that he can clearly notice that he is hearing a voice without the mouth of the shadow moving.
Assuming you are talking about the allegory of the cave: (it is an allegory after all, the people watching the shadows might as well just turn their heads around)
Here it is easy to see that the ears alone will do the work. There is no real physics experiment, the listener naturally can detect the origin of the sound. But assuming he was born in that cave and is not accustomed to any other sound then an experiment to determine the truth can take many forms:

Hold up an object and obstruct the shadow which will show the voices are independent of the shadows. Or maybe shine a light on the wall that might make the shadows disappear.
Talk at the same time and try to notice the discrepancies and similarities between the person's voice and the shadows

I am pretty sure there are some other ways but all are based on intuition since we are assuming that this person in a cave has always been in that cave and that he does not have any equipment or technological resources.
Answer: The physicist should want to demonstrate whether talk and shadows have a necessary relationship, or that they are independent.
If you can do experiments things are fairly easy. As RetroCausalElectron points out, holding up objects to (1) make a shadow or (2) hide a talking shadow demonstrates that (1) shadows on the wall do not necessarily talk and (2) just because a shadow looks like a non-talking shadow doesn't mean it isn't talking. So the physicist would at least conclude that shadows he could make have no effect on talking.
A sceptic in the cave would point out that maybe cave-people don't have the right shadow technology to make talking shadows. However, the physicist can also talk and generate shadows, producing a talking shadow that obviously was causally generated by the cave-person. A platonist might suggest that maybe the physicist is a shadow somehow projecting a 3D person into the world, but the physicist would note that he still exists when he is in the shade and has no shadow.
If the cave-people cannot cast shadows (they are chained below a ledge unable to access the light above) then the whole issue is observational physics and theory formation. Gathering data will show many cases of talking and non-talking shadows.
Careful gathering of data will also reveal differences in volume correlated with size and diffuseness, and a really astute physicist would notice that the delay between something happening to the shadow (a shadow horse kicking a shadow person) and the sound (a bang and a curse) exists, and is also correlated with the size and diffuseness. If the physicist can make sounds he might also recognise similar delay effects in echoes in different directions, giving a hint at what is going on. A possible hypothesis is that shadows generate delayed/weakened sounds with an effect size linked to the size/diffuseness property, another possible hypothesis is that some separate thing causes both shadows and sound and has a property that affect both observations. Which one is simplest and most testable?
A counterexample to the first hypothesis would be a sharp shadow with a long-delayed sound. More research will demonstrate that the delay and volume relation is imperfect, but for particular kinds of shadows like the philosopher-shadows or horse-shadows they hold relatively well. There are quiet shadows like the downcast slave-shadow that are low volume even when very sharp, and loud shadows like the demagogue-shadow that are loud even when diffuse. So the size/sharpness to volume relation appears to be a weaker explanation than assuming there are two parameters for individual shadow types, "distance" that determines a scaling of the basic shadow size and basic volume, and loudness that scales the volume but depends on type.
At this point the sophist complains that with that many parameters the physicist can fit anything to his data, and the physicist says he just has made a useful model. The philosopher next to them will complain about assuming unseen abstractions and the problem of induction. Meanwhile the engineer long ago picked the lock on his chains and escaped, none the wiser.
Comment: So as i understand your answer, physicist can't proove that shadows aren't talking? If there is no correlation between diffuseness and loudness, so what? Shadows are something "in the wall" and the sound happens only when the shadow comes. Loundness and sharpness are separate parameters of that shadow (like for example, temperature and impulse of the ball. We don't say that there is something which cause temp and impl.)  There is no sound in the cave by itself. Yes, some shadows are silent. But that doesn't proove that shadows aren't talking, it just proves that sound does not create shadows.
Comment: Even if a physicist introduce a lot of parameters that describe shadow and sound, i still can't understand why he should think that somethng external cause shadow and sound. All the correlations will be concieved as the properties of that dark thing "on the wall". I think the statement "that dark thing on the wall is the source of the sound" will be declared as a physical law.


2.5 inch laptop hard drive in desktop?

Question: I have a Toshiba MQ01ABD032 250gig laptop hard drive. I was wondering if I could hook it up to my desktop pc. Will it be fried from the 12volt line on the sata power connector?
Comment: Possible duplicate of [Can a laptop hard-drive be used in a desktop machine?](https://superuser.com/questions/451079/can-a-laptop-hard-drive-be-used-in-a-desktop-machine)
Comment: @music2myear Only marked as duplicate, based on the title of the question.  Other considerations, such as power, aren't necessary to delve into because of the standardization of SATA.  No aggression meant, however.
Comment: No aggression perceived. As the linked Q only mentioned SATA in passing it could be easy to miss. I noted this and thought this question was just different enough to be not actually duplicate.
Comment: Marked dupe question does not deal with power, which is the specific point of this Q.
Comment: @music2myear, the other asks "can you do it?"  Power isn't specifically addressed, but if it was an issue, wouldn't that be mentioned in the answers as a problem in trying to do it?  This question only asks whether power is an issue, so it seems covered by the other.  If it asked "why" it is or isn't an issue, that would seem like more of a different question.  Good answer, BTW.
Answer: No, your drive will not fry on the desktop SATA power connector.
The SATA standard is designed to handle this gracefully. The 15-pin SATA power connector contains 3.3V, 5V, and 12V connectors, and the device will connect to the appropriate pins depending on its needs.
Pins

3.3V Power
3.3V Power
Enter/Exit Power Disable mode
Ground
Ground
Ground
5V Power, Pre-Charge
5V Power
5V Power
Ground
Staggered spinup/activity
Ground
12V Power, Pre-Charge
12V Power
12V Power

The 2.5" drive, only needing 5V, will only connect to the necessary pins.
For more reading, Wikipedia's article could be a good start: https://en.wikipedia.org/wiki/Serial_ATA#Power_connectors
Comment: Based on good advice from @JamieHanrahan this answer has been substantially modified. The binary answer did not change, but the explanation of why and the background has been corrected.

2.5 inch laptop hard drive in desktop?


search query "alien vs predator"

Question: How do you do so that when you search for "alien vs predator" you also get results with the string "alienS vs predator" with the "S"
example http://www.torrentz.com/search?q=alien+vs+predator
how have they implemented this?
is this advanced search engine stuff?
Answer: This is known as Word Stemming.  When the text is indexed, words are "stemmed" to their "roots".  So fighting becomes fight, skiing becomes ski, runs becomes run, etc.  The same thing is done to the text that a user enters at search time, so when the search terms are compared to the values in the index, they match.
The Lucene project supports this.  I wouldn't consider it an advanced feature.  Especially with the expectations that Google has set.
Answer: Checking for plurals is a form of stemming.  Stemming is a common feature of search engines and other text matching.  See the wikipedia page: http://en.wikipedia.org/wiki/Stemming for a host of algorithms to perform stemming.
Answer: Typically when one sets up a search engine to search for text, one will construct a query that's something like:
SELECT * FROM TBLMOVIES WHERE NAME LIKE '%ALIEN%'

This means that the substring ALIEN can appear anywhere in the NAME field, so you'll get back strings like ALIENS.
Answer: When words are indexed they are indexed by root form. For example for "aliens", "alien", "alien's", "aliens'" are all stored as "alien".
And when words are search search engine also searches only the root form "alien".
This is often called as Porter Stemming Algorithm. You can download its realization for your favorite language here - http://tartarus.org/~martin/PorterStemmer/
Answer: This is a basic feature of a search engine, rather than just a program that matches your query with a set of pre-defined results.
If you have the time, this is a great read, all about different algorithms, and how they are implemented.
Answer: You could try using soundex() as a fuzzy match on your strings.  If you save the soundex with the title then compare that index vs a substring using LIKE 'XXX%' you should have a decent match.  The higher the substring count the closer they will match.
see docs: http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_soundex

search query "alien vs predator"