Swift: What is the best way to quickly search through a huge database to find relevant result?

Question: I'm trying to implement a search algorithm that can search through hundreds of thousands of products and display the most relevant searches.
My current process is

Get user's input and filter out prepositions and punctuations to arrive at keywords

Break keywords into and array

For each of the keywords find all the products that contains the keyword in the product description and add all the product to a RawProductDictionary.

Calculate Levenshtein Distance number between the Keywords and each product description.

Create an array of product based the Levenshtein Distance number.

this question builds on top of this question
Swift: How can the dictionary values be arranged based on each item's Levenshtein Distance number
this is my Levenshtein Distance function
<code>  func levenshteinDist(test: String, key: String) -> Int {
   let empty = Array<Int>(repeating:0, count: key.count)
   var last = [Int](0...key.count)

   for (i, testLetter) in test.enumerated() {
       var cur = [i + 1] + empty
       for (j, keyLetter) in key.enumerated() {
           cur[j + 1] = testLetter == keyLetter ? last[j] : min(last[j], last[j + 1], cur[j]) + 1
       }
       last = cur
   }
   return last.last!
 }
</code>
This is the function that implements step 5
<code>   func getProductData(){

  Global.displayProductArry = []
  var pIndexVsLevNum = [String : Int]()
  for product0 in Global.RawSearchDict{
      let generatedString = product0.value.name.uppercased()
      let productIndex = product0.key
      let relevanceNum = levenshteinDist(test: generatedString, key: self.userWordSearch)

      pIndexVsLevNum[productIndex] = relevanceNum
  }

    print(pIndexVsLevNum)
    Global.displayProductArry = []

    for (k,v) in (Array(pIndexVsLevNum).sorted {$0.1 < $1.1}) {
        print("\(k):\(v)")
        Global.displayProductArry.append(Global.RawSearchDict[k]!)
    }
}
</code>
The code works but the products are not that relevant to the user input

Levenshtein Distance number is not always indicative of relevance. Products with shorter description are usually disadvantaged and missed.

what is the best way to implement searching through hundreds of thousand of products quickly in swift?
Comment: Depends on so many aspects of the project like Datasets, Data storage, Score system, business limitations, user expectations, etc.. There is no **best way** for this.
Answer: I believe you are looking for Full-Text Search.
You could use existing tools for that, rather than creating your own information retrieval process.
Looks like SQLite can give you that:
See: https://medium.com/flawless-app-stories/how-to-use-full-text-search-on-ios-7cc4553df0e0
Answer: According to Wikipedia:

Informally, the Levenshtein distance between two words is the minimum
number of single-character edits (insertions, deletions or
substitutions) required to change one word into the other.

You should be using Levenshtein distance to compute individual words with each other, not entire product descriptions with a single word. The reason you would compare individual words with each other is to determine if the user has made a typo, and determine if he actually meant to type something else. Hence the first part of your problem is to first try to clean up the users query.

First check for perfect matches against your keyword database
For words which do not match perfectly, run Levenshtein to create a list of words  most closely matching.

Lets stepback for a moment and look at the big picture:
Simply using Levenshtein distance by itself, is not the best way to determine which is the most relevant product by comparing with the entire product description, since normally the product description will be much much larger than a users query and will describe a variety of features. Let us assume that the words are correctly spelled and forget spellchecking for a moment so we can focus on relevancy.
You will have to use a combination of techniques to determine which is the most relevant document to display:

First, create a tf-idf database to determine how important each word is in a product description. Words like and, is, the etc are very common and usually will not help you determine which document is most relevant to a users query.
The longer a product description is, the more often a word is likely to occur. This is why we need to compute inverse document frequency, to determine how rare a word is across the entire database of documents.
By creating tf-idf database, you can rank the most important words in a product description, as well as determine how common a word is across all documents. This will help you assign weights to the value of each word.

A high weight in tf-idf is reached by a high term frequency in a given
document, and a low document frequency of the term in the whole
collection of documents.

Hence, for each word in a query, you must compute the relevancy score for all documents in your product description database. This should ideally be done in advance, so that you can quickly retrieve results. There are multiple ways you can compute TF-IDF, so based on your ability, select one option, and compute for TF-IDF for every unique word in your document.
Now how will you use TF-IDF to produce relevant results ?
Here is an example:

Query: "Chocolate Butter Pancakes"

You should have already computed the TF, and IDFs for each of the three words in the query. A simplistic formula for computing relevance is:
<code>Simplistic Product Description Score: TF-IDF(Chocolate) + TF-IDF(Butter) + TF-IDF(Pancakes)
</code>
Compute the Product Description score for every single product description (for the words in the query), and sort the results from highest score to lowest score to get the most relevant result.
The above example, is a very simple explanation of how to compute relevancy, since the question you asked is actually a huge topic. To improve relevancy, you would have to do several additional things:

Stemming, Lemmatization and other Text Normalization techniques prior to computing TF-IDF of your product descriptions.
Likewise, you may need to do the same for you search queries.

As you can imagine, the above algorithm to provide sorted relevant results would perform poorly if you have a large database of product descriptions. To improve performance, you may have to do a number of things:

Cache the results of previous queries. If new products are not added / removed, and the product descriptions do not change often, then it becomes much easier.
If descriptions change, or products are added / removed, you need to compute TF-IDF for the entire database again to get the more relevant results. You will also need to trash your previous cache and cache new results instead. This means that you would have to periodically recompute TF-IDF for your entire database, depending on how it often it is updated.

As you can see, even this simple example is already starting to get complicated to implement, even though we haven't even started talking about more advanced techniques in Natural Language processing, even things as simple as how to consider the usage of synonyms in a document.
Hence this question is simply too broad for anyone to provide an answer on stackoverflow.
Rather than implement a solution yourself, I would recommend searching for a ready-made solution and incorporating it in your project instead. Search is a common feature nowadays, and as there are many solutions available for different platforms, perhaps you could offload search to a web-service, so you are not limited by having to use Swift - and can then just use ready-made solution like Solr, Lucene, Elastic Search etc..


how much time is polite for the asker to be 'outside' of Stack Overflow once a question was posted?

Question: Considering that a lot of questions got good answers just a few minutes after the question, and that some people need more information for better understanding and better answers: how much time is polite to be 'outside' of Stack Overflow once a question was posted and then return back to check answers and comments in your question?
Comment: Fourteen seconds.
Comment: Six to eight weeks?
Comment: I try to check back several times in the first hour, but the behavior is self-correcting: people who persist in abandoning their questions for long periods will get less satisfactory results. Then they will either learn better or go away...
Answer: You get the Fastest Gun In The West answers for at minimum the first 5 minutes after you post.  Usually a crappy version, quickly typed to get lazy upvotes or to lock-in the Enlightened badge.  Meanwhile, the answerer still works on typing a real answer.  Give them another 5 minutes in case they were a bit late to seeing it.  Posting a comment that says "what do you mean" is a waste of your time, wait 10 minutes.
Find a sword to pass the time.
When fifteen minutes have passed and you still didn't get a meaningful answer then you've got a problem with your question, it isn't good enough.  Jump in and start editing your post, you'll need to give a lot more details.  You have however entered the very long tail of SO, your edit can only bump it up-hill a bit.  The odds of a sugar daddy showing up to cast the spell are low.  Start looking around for somebody else's question to answer so you have enough rep to put a sizable bounty on yours.
Comment: Besides drowning us in your bitterness, this doesn't even answer the question. He wants to know how often people should be expected to check in and answer comments or address confusion, not how fast they should expect an answer to come in
Comment: I don't get the bitter and sad comments.  I'm pretty sure that a significant majority of SO users like it just fine and wouldn't have it any other way.  Well, except the bit about not getting an answer within 15 minutes.
Comment: +1 for cynical realism :D
Comment: Although, looking at the update, I wouldn't see it as bleakly. I frequently see questions first answered three, five, seven hours after asking.
Comment: +1 @Michael Mrozek, you are right: the question is not about speed in the answers. I found very interesting all this comments, btw.
Comment: A long tale but a sad one...
Answer: Difficult to tell. Very often, when people ask questions, they will return after a few hours to check on the progress.
It also depends on how badly you need an answer, I guess. A comment that doesn't get answered quickly will simply be forgotten by users, until you respond.
Comment: This *"return after a few hours to check on the progress"* time scales is entirely appropriate for many forums on the internet, and is usually a sign of a user unfamiliar with SOFUE.


In the Dutch Defense, why does white play b3 and Ba3?

Question: In the Stonewall Dutch Defense, why does white play b3 and Ba3? Should I try to avoid it, or does it not matter?
<code>[fen ""]

1. Nf3 e6 2. d4 f5 3. g3 Nf6 4. Bg2 d5 5. O-O Bd6 6. b3 O-O 7. c4 c6 8. Ba3
</code>
Answer: The main reason is to exchange black's good bishop, and what would probably end up being white's bad bishop after a later e3. Another common way of doing this is Bg5xNf6. This also highlights black's dark-square weakness problem that he will have to watch out for for quite a while.
This exchange on a3 leaves black's other bad bishop on c8 more markedly bad, and the e5 square more weakened. After the exchange on a3, black can, and usually should, gain a tempo with Qe7, and the white Na3 will redirect to Nc2-e1-d4 (or sometimes via Nc2-b4-d3) to further control e5.
This scores well in my database at 67% for white, but many of the games are between players of disparate ratings. At the higher levels, it seems that black can hold equality, so I would say that if you are stronger, there is no need to avoid it, but if you are weaker against a good positional player, you might want to try to get in Qe7 earlier to make Ba3 harder, but white still has a4 then. I am not sure you can really, and totally avoid it if that is the plan white chooses.
Comment: Thank you Phishmaster..
Comment: I hope this helped you.
Comment: Thanks for trying. I am not sure what the trigger I missed was, but I did not get that final hat. I appreciate it.
Answer: In the pawn structure of the Stonewall Dutch the light squared bishop is the sick piece in black's position and the dark squared bishop is the star. If white can exchange the dark squared bishops then white already stands much better because of all the resulting weak dark squares in black's position, particularly e5.
Consequently it is a mistake for black to reply to 6. b3 with O-O allowing Ba3. Essential is Qe7 to prevent Ba3 and the subsequent trade of black's best piece. An early b6 is probably also important for black to allow the plan of a4 followed by Ba3 anyway by white to be answered by c5 blocking the exchange, possibly with Nbd7 to reinforce c5.
Comment: I don't know that I would call it a "mistake". The game I referenced, black was GM Francisco Vallejo Pons, who peaked at 2724. I doubt he would make a mistake on move 6. Besides, white can still force it with a4 and then Ba3. It is just a game as we say.
Comment: Thank you also Brian.
Answer: When you look at the position, you see a horrible light Bishop for Black, and you see a dark Bishop for White that doesn't have a whole lot to do, effectively. It's biting on the d-pawn, its 'natural prey' (the Black Knight on f6) cannot be effectively pinned, as the queen will be wanting to slide to e8 in this position anyway, so what's it going to do?
The idea of trying to force the exchange of Black's dark Bishop by effectively pinning it against the Rook is very attractive. It's more effective against the old stonewall lines, when the Bishop is on e7, because in this line Black can, effectively, get a free Queen move in by moving it to e7, forcing White into spending another tempo to achieve the aim.
It's a worthy aim, simply because of how mind-achingly bad Black's other Bishop is. But it's not a winning aim. Black can get some free development playing off the misplaced Knight on a3, and where's it going to go? It can't meaningfully get to b5, or really even c4. Back home to b1 or c2 aren't really attractive, and the only other choices are something like Qc1 or letting go of the tension in the center with c5. (It should be noted that the central tension is one of White's weapons against Black's buildup for a wing attack, so dropping it is kind of a big thing; I've been known to try and tease White into doing that with ...Qd6 simply because of how much stronger that makes an eventual ...e5. Depends on how you feel about double-edged positions whether you'll like that idea. White can get an interesting steamroller going on the Queenside if Black isn't paying attention.)
The best thing I can say for it is it can be effective against a lot of Dutch players who haven't spent time thinking the unthinkable: How do they attack without their chief weapon, the Kingside Bishop? Having to answer that question can throw a lot of Stonewall players off their game.
On the other hand, it does relieve one of the major issues with the Dutch, which is that Black has one more minor piece than they have room for, and that extra space can be handy as the pieces roll out. If Black has learned to be flexible, their game still has a lot of play in it (one reason why this isn't seen more often in high-end play).
If you're looking for a good anti-Dutch line, I've always felt the lines based around Nh3/Bf4 offer more than the b3/Ba3 ones. YMMV.


Is post fit simulation using timing netlists necessary although design meets setup-hold time requirements?

Question: In a purely synchronous design, if the design has positive slack for setup and hold times, it means that it meets timing. Therefore, provided that we have carried out static timing analysis and are happy with the slack at our targetted frequency, is there any point in carrying out post fit simulation with timing netlist anymore?
Comment: Generally, if static timing passes, you're good. Exceptions : issues with I/O signal timings (on the device pins), and chasing suspected tool (synthesis, P&R, or static timing analysis) bugs.
Answer: Yes, post-fit timing analysis using timing netlists is necessary, assuming that you really want your design to work.
Remember that all simulations are just an approximation of the actual device behavior. Going through the process of fitting your design can change some of the routing and wiring length, because the tools don't really know how far apart your logic cells are until after they are placed. Hence, the parasitic capacitance and resistance of the wiring can change, leading to changes in propagation delay.
Every step in the design process leads you closer to the final product, so a timing analysis should be performed after each step using delay values that are extracted from the actual design. After every step your timing analysis will become more accurate because it more correctly represents the actual design.
You should always create a testbench that will check whether the design still meets all of its functional requirements. Ideally, the testbench will spit out an error message if the current simulation results differ from the previous simulation results. After every change or improvement in the design you repeat the simulation...this is called regression testing.
Comment: Looks like I misunderstood your question. If you have properly constrained your input signal timing, required output timing, and clock timing then a static timing check will give you a high level of confidence. Nevertheless, if you are committing an ASIC to silicon you should perform every conceivable simulation before going to fab. The analysis tools are not perfect.
Comment: But if the static timing analysis says  that I have worst case positive slack at target click frequency in post fit netlist (which includes real worst case delays as far as I know), I thought that should be enough.


Creating a Grid in WGS 84 in Feet

Question: I am using Using QGIS 18.16 to try and create a grid measured in feet.
I am having some issues with the best way to create a accurate grid in a project projected in WGS 84.  I know that it uses degrees but i need to come up with a way to create it in feet.  I have tried the Vector Grid creation and a few plugins but i feel like i am missing something and making this a little to hard.
Answer: WGS84 coordinates are coordinates on a spheroid. You can't have a regular grid on a spheroid.
Over a small area you can use another coordinate system that is planar and make a regular grid with that, but it won't be a regular grid if you transform it to WGS84 (lat-long) coordinates.
Comment: But be aware that a *grid* created in a projected coordinate system will not be regular when viewed/projected to geographic coordinates.. my advice is to make your map in a suitable projected coordinate system otherwise your grid will look funny and your scale bar would be unusable at best.

Creating a Grid in WGS 84 in Feet


.htaccess removes trailing periods from query

Question: Having some trouble where .htaccess is removing the trailing period from strings. I am running Apache/2.2.12 (Win32).
This is not a production system, so there is nothing else in .htaccess other than:
RewriteEngine on

...and rules, and this works well in all cases so far, unless there is a trailing period in the name. Rules all take the form of:
RewriteRule ^thing/(.*)$ showthing.php?thingname=$1

A link on A.php:
 ...CLICK

So in showthing.php, I simply do:
$sTHINGname= !isset($_REQUEST['thingname']) ? NULL : $_REQUEST['thingname'];

...then $sTHINGname is run though the db to find the ID.
The trouble is that when thingname has a trailing period (e.g.: "M.O.X.")
echo $sTHINGname;  // shows M.O.X, which does not match an ID

One solution is to do:
str_replace( '.', '%2E', $name );

But when the links are placed in the referring page (A.php), and this works, but the links read:

...which is a bit inelegant and does not solve the problem, because if a user types as a URL "...thing/M.O.X." they still don't get a result.
I checked out replacing $_REQUEST with $_GET, but the result doesn't change. Also:
RewriteRule ^(thing/)(.*)$ showthing.php?thingname=$2

...and variations all show the same results.
Most search results dealing with htaccess and periods are domain issues, so I wonder if anyone has any ideas about this one?
Comment: Do you have a `[L]` on that `RewriteRule`?
Answer: Try this:
RewriteRule ^thing/(.+)$ showthing.php?thingname=$1 [L,QSA]

Comment: (via OP): thanks for suggesting. [L] has no effect, and neither does [NE]
Comment: The change was `*` to `+` in addition to the flags. I didn't only add the flags.


Why Does Light Not Become Polarized In A Magnetic and/or Electric Field?

Question: I am familiar with the Faraday Effect, but I remain confused as to why the electric and/or magnetic components of light do not naturally align themselves with a magnetic or electric field (in a vacuum) and thereby become polarized.
Comment: I suspect the answer to this is because electromagnetic waves are linear.  i.e, light doesn't interact with electric and magnetic fields; light passes through light, light passes through electric fields and magnetic fields.
Comment: Just by superposition of waves: if you have an electric or magnetic field on a region of space and light goes through it, within that region the fields will sum to each other, and be the same as in the beginning when the leave that region.
Answer: The Faraday effect is dispersive, so it is coherent and reversible. It tweaks the evolution of the light but doesn't cause "relaxation". In general, a system cannot reach a ground state, like becoming aligned to a magnetic field, without some sort of dissipation (e.g., absorption).
The linear algebra way to think about this is as follows: consider orthogonal states of light, such as $H$ and $V$ linear polarizations. After coherent evolution, they remain orthogonal, or else the evolution would not be reversible (given the final state, the initial state can be uniquely determined). This logic shows us the range of possible options for coherent evolution. The polarization can rotate (like the Faraday effect), it can go from linear to elliptical or circular (like a quarter waveplate), or it can undergo a mirror reflection (like a half waveplate). Of course, any combination of those options is also valid. But it can't align without ''losing information'', which would require a absorptive polarizer. (What I've just described is the $U(2)$ group of unitary matrices on $2\times 2$ complex vectors.)
Note: when I say "reversible", I mean coherent or unitary. I'm not referring to time reversal symmetry, which the Faraday effect breaks.
Edit
If we're looking for a way for light to align to the magnetic field, we need the information on the original polarization of the light to be lost. The Faraday effect is off-resonant and cannot achieve that. One way to realize this is through dichroism, where one polarization (usually circular) of light is absorbed. This process is clearly not coherent/unitary because it erases the information on what the incoming polarization is. After absorption, with the right setup, the light can become aligned with the magnetic field. (I can give an explicit example with atoms, if you'd like.)
Sorry if this repeats my comment.
Comment: I guess the next logical question is: why does polarization require dispersion?
Comment: What do you mean by "relaxation"?
Comment: The Faraday effect in particular is an off-resonant excitation, hence it's only dispersive. Dichroism is a related effect in which the light polarization it changed or rotated through absorption -- this is what I mean by "relaxation". Incidentally, dichroism is an excellent example where the light becomes aligned with the magnetic field: one ciruclar polarization is absorbed and the other one remains. "Information" about the original state is lost. To distinguish these effects, I'm referring to the way that light can be dissipitated, absorbed, or incoherently scattered.
Answer: Light does become polarized in the presence of a magnetic field. That is how some 3-dimensional maps of the magnetic field lines above certain surface structures of the Sun are recovered, by means of observing the direction of polarization of light of a specific infra-red line, that arrives from the photosphere. I remember it was a paper in Nature by Krupp and Collados and others. It is a relatively recent technique (from the last decade, I think) because it was technically difficult to observe the polarization of light from the Sun until recently.
See this link for the paper (it is really a letter). A popular description of the technique can be found here.
Comment: It is unclear from my brief scanning of the article whether the magnetic field was measured for within solar flares. According to the other answers it must have been within the solar flares.
Answer: @QuantumDot You seem to be close to the answer to this question. Neither the electromagnetic field Lagrangian nor the gauge theories of fundamental interactions include interactions between photons and electric or magnetic fields. An electric or magnetic field, on their own, represent very low energy photons. To say that a photon interacts with a magnetic or an electric field would suggest that photons interact among themselves. Photon-photon interactions are possible via two fermion 'square' loops, like e- e+ for example. Furthermore photons have neither electric nor magnetic dipole moment. Hence your suggestion that photons pass through an electric or magnetic field undisturbed is right. I have increased your comment by 1.
Comment: @JoeHobbit That is correct. The electromagnetic force (that is magnetic and electric fields) couple only to electric charges or magnetic poles in the form of dipoles such as the magnetic compass for example. This is why neutrinos are hard to detect as they do not interact with the electromagnetic field. Also thing of the way charged particles such as e+ and e- are detected in bubble chambers, taken by the strong magnetic field in opposite circular motion and give the famous V-shape tracks. I hope this helps?
Comment: Are you saying that a magnetic dipole and/or an electric monopole is required to interact with an electric or magnetic field? How can magnetic fields (in EM waves) exist without dipoles?
Comment: One thing separated your answer from the others: Yours mentioned that a dipole/monopole, which is lacking in light, is required to interact with an electric or magnetic field. I almost chose anna v's answer for clarity regarding the exceptional event of Delbruk Scattering.
Comment: @JoeHobbit Many thanks for that. The point is that we are all learning something, and that is by far more valuable.
Comment: @John, if we're discussing the Faraday effect, it's completely incorrect to say that there's no interaction. The whole point is that you can use materials to couple DC magnetic fields and optical fields. If you restrict yourself to a vacuum, you miss most of the interesting phenomena in electromagnetism and I'd be out of a job.
Comment: @emarti Thanks for your comment. In the presence of a material of course light can become polarised, and it does not need an external magnetic field for this, as with polarising filters. However, you are right, in the presence of matter the magnetic field sets off other effects in the material, a solid say, known from standard EM theory and they are interesting. The discussion was focussed on the point whether a magnetic field on its own can polirise light. In this case one can consider very strong magnetic fields in which $e^+-e^- pairs can be created. This has interesting effects. Regards
Answer: I think the crux of the matter lies in your :

why the electric and/or magnetic components of light do not naturally align themselves

What does "align" mean other than an interaction?
So the question  "do photons interact with an electric or magnetic field in vacuum"? has to be answered first before going into the details of the type of interaction.
The answer to this last is "yes, with very very small probability". It is called Delbruck scattering and has been experimentally observed.
It is a QED only effect, it does not exist in classical electromagnetic theory and the measurements are used as an experimental confirmation of QED. It is evidence of vacuum polarization in the strong electric fields of nuclei.
So the photons do interact with the fields through higher order QED diagrams, and not simple higher order, the lowest order allowed is sixth order, which makes this interaction very improbable. It is small in the strong fields around nuclei, it is infinitesimal in the distant coherent fields  of atoms, which is what an electric or magnetic field is in vacuum. To have any fields you need the electric and magnetic sources of atoms.
The "infinitesimally low probability of interaction" is what negates the "natural" in your question. A beam of photons does not align its electric or magnetic field to the direction of electric and magnetic fields in vacuum because the interaction probability is practically non existent.
Comment: You are right anna, such effects are possible and measurable. However, they have been mentioned in the discussion by John, in the context of vacuum polarisation. I feel your detail gains you +1.
Answer: I think, from a viewpoint of photonics, light, or photon, can only interact through charge and gravity. View light as bosonic vibration of space-time. Charge is the source for such vibrations and responsible for scattering, absorption as well. Gravity affect light because gravity is the curvature of space-time itself.
From a pure wave picture, any effect of light is tracked down to the effect of wave-interference: The scattering is the wave generated by dipoles inside matter excited by incident wave; absorption is simply the scattered wave out of phase with respect to incident one, nonlinearity is anharmonic scattering, etc, etc. Wave themselves simply add up by principle of interference without changing each other's internal property. Polarization is such a property. Hence light add up with another light will not change either's polarization, although they may form a standing wave or beat.
Comment: Er...no. A *"vibration of space-time"* would be a gravitational wave and if you wanted to view it in quantum terms you would look for gravitons. A photon is an excitation of the electromagnetic field.
Comment: yet what is a "electromagnetic field".Explaining photon as electromagnetic field firstly, can't explain why EM wave and Gravity waves have the same speed invariant of space-time; secondly, it is simply a circular argument, essentially explaining nothing. By thinking of photons as bosonic vibration on space time (similar to that of string theory), helps at least on an intuitive basis to understand its superposition, and propagating nature. For the full theory, Check xiaogang wen's spin net theory. (at MIT)
Answer: Light does become polarized in a magnetic field. The magnetic field of a black hole was detected due to the polarization of light. Check this article:
http://www.iflscience.com/space/black-holes-powerful-magnetic-field-observed-first-time


Removing a file in /etc/grub.d

Question: Can I remove a file in /etc/grub.d that was not created by me without it being overwritten? I'm using fedora if that matters.
Answer:
Can I remove a file in /etc/grub.d?

You can remove the files under /etc/grub.d (it doesn't harm the current configuration), but it is better to keep them it will be used by grub to correctly set the grub.cfg after executing grub-mkconfig or updating grub, there is no benefit to remove them.
here is the content of /etc/grub.d/Readme (From Debian):
All executable files in this directory are processed in shell expansion order.

  00_*: Reserved for 00_header.
  10_*: Native boot entries.
  20_*: Third party apps (e.g. memtest86+).

The number namespace in-between is configurable by system installer and/or
administrator.  For example, you can add an entry to boot another OS as
01_otheros, 11_otheros, etc, depending on the position you want it to occupy in
the menu; and then adjust the default setting via /etc/default/grub.

ls /etc/grub.d:
00_header        10_linux      30_os-prober      40_custom  README
05_debian_theme  20_linux_xen  30_uefi-firmware  41_custom

The must known /etc/grub.d/40_custom file is used to manually adding a custom grub entry to grub.cfg.

Removing a file in /etc/grub.d


Is there a difference between a bit mask and a bit array?

Question: I have heard the two terms used interchangeably.  Is there a difference?
For example,
unsigned char chessboard : 64; /* Bit mask */
unsigned char chessboard_2 [64]; /* Bit array */

Answer: None of them is a bitmask. The first is the definition of a bitfield which should only be valid as a struct member and the second is an array of 64 unsigned chars.
Comment: Thanks @Swordfish.  What is an example of a `bit mask`, though?
Answer: Bit Mask
A bit mask is a binary value that's used to refer to specific bits in an integer value when using bitwise operators. For instance, you might have:
unsigned int low3 = 0x7;

This is a bit mask with the low order 3 bits set. You can then use it to extract a part of a value:
unsigned int value = 030071;
unsigned int value_low3 = value & low3; // result is 01

or to update part of value:
unsigned int newvalue = (value & ~low3) | 5; // result is 030075

Bit Array
A bit array is an unsigned integer, or an array of unsigned integers, that's used to hold a sequence of boolean flags, where each value is in separate bits of the integer(s). If you have lots of boolean values to store, this is saves lots of memory compared to having each of them in a separate array element.
However, there's a tradeoff: in order to access a specific flag, you need to use masking and shifting.
If your bit array is small enough to fit in a single integer, you might declare:
uint32_t bitarray;

Then to access a specific element of it, you use:
bitvalue = (bitarray >> bitnum) & 0x1;

and to set an element:
bitarray |= (1u << bitnum);

and to clear an element:
bitarray &= ~(1u << bitnum);

If the bit array needs multiple words, you declare an array. You get the array index by dividing the bit number by the number of bits in each array element, then use the remainder to determine the bit number within that word and use the above expressions.


CPU serial number

Question: How do I obtain the serial number of the CPU in a PC?
Comment: Are your CPU's Pentium III's?  If not, is there a different question you are actually trying to ask?
Comment: This page has a great answer. http://stackoverflow.com/questions/3474940/unique-computer-id/3474966#3474966
Comment: http://stackoverflow.com/questions/3474940/unique-computer-id/3474966#3474966
This page has the answer. :)
Answer: I have the ultimate answer for this without any external libraries. Simply type this:
wmic bios get serialnumber
This will give you the Serial Number on the PCs chassis ;)
(found in microsoft's knowledge base)
Regards!
Comment: With my board it returns "To be filled by O.E.M." => not reliable => probably better to use something else, like the MAC address if available.
Comment: returned the text "System Serial Number" for me on my i7. Do you have the link to the mentioned knowledge base article?
Comment: The text "System Serial Number" is typical in white box systems. It is supposed to be replaced by the PC maker before they sell it.
Comment: Returns the Tag Number on a Dell.
Answer: Remember that most computers these days ship with CPU ID disabled in the BIOS. See CPUID on Wikipedia
Answer: There is no CPU serial ID (PSN; CPUID edx bit 18 "psn"  Processor Serial Number) after Pentium III in Intel CPUs; and there was never any psn in AMD chips:
https://software.intel.com/en-us/forums/watercooler-catchall/topic/308483  (at 2005)

However, keep in mind that only the Pentium III Xeon, Mobile Pentium III and Pentium III processors support the processor serial number feature introduced by the Pentium III processor. No other Intel processor supports the processor serial number feature

https://en.wikipedia.org/wiki/Pentium_III#Controversy_about_privacy_issues
https://en.wikipedia.org/wiki/CPUID#EAX=3:_Processor_Serial_Number

EAX=3: Processor Serial Number
  See also: Pentium III § Controversy about privacy issues
This returns the processor's serial number. The processor serial number was introduced on Intel Pentium III, but due to privacy concerns, this feature is no longer implemented on later models (the PSN feature bit is always cleared). Transmeta's Efficeon and Crusoe processors also provide this feature. AMD CPUs however, do not implement this feature in any CPU models.
Answer: Even with CPUID enabled is there actually a serial number available in modern processors? I remember there being a big outcry in the Pentium 3 days when this whole serial number issue was raised.
Comment: There is no CPU serial ID (PSN) after Pentium III in Intel CPUs: https://software.intel.com/en-us/forums/watercooler-catchall/topic/308483 "However, keep in mind that only the Pentium III Xeon, Mobile Pentium III and Pentium III processors support the processor serial number feature introduced by the Pentium III processor. No other Intel processor supports the processor serial number feature:" (at 2005); https://en.wikipedia.org/wiki/Pentium_III#Controversy_about_privacy_issues
Answer: This is and old thread. But I had a same problem, but I got the following logic working without too many ifs, ands or buts.
The problem with CPU serial number is that it does not always work in virtualized environment.
I did the following logic with a set of Windows-based servers:
Win32_BIOS can provide you a serial number of the bios. We need to keep in mind that if the system is virtualized, you could end up with same bios serial number for all servers.
Win32_NetworkAdapter can provide you a MAC that you can use as well. In cases where you have multiple NICs, you will end up with multiple-MACs.
Combining both these IDs, I had all unique set over a set of 6000 servers spanning across physical and virtual. This was really simple to implement using ManagementClass & ManagementObject.
But just a caveat: when you try to get the MO instance remotely, it'll take more than a few seconds on a <5ms latency 10Gbps optical network. So if you do the math it took me over 3 hours on a single thread operation. Since this is more like a low-priority traffic I didn't want to spam my network for gathering WMI data on multi-threaded call.
Answer: Ivy Bridge CPUs and newer all include a PPIN (Protected Processor Identification Number). Access to this feature can be blocked by the computer's firmware.
https://lore.kernel.org/patchwork/patch/736614/
Answer: Executing the CPUID instruction with the proper register settings will retrieve the processor serial number in EAX, EBX, ECX, and EDX.  However, this functionality is only available on Pentium 3 and later processors.  Also on Pentium 4 and newer processors the instruction always returns 0x00000000 in all 4 registers.  Later model Pentium 3's may also return 0x00000000's.  The feature was primarily aimed at copy protection, allowing software to be linked to specific processors.  It did not go over well with the community, and lawsuits ensued.  The feature was removed from late model P3's and all newer processors.  The feature is present in newer processors for compatibility reasons.  it is rumored than you can special order processors with serial numbers, btu the minimum purchase is something like 1 million processors.  For the specific register settings prior to executing the CPUID instruction, check Intels system programmer PDF available through their website.
Also -

#include <Windows.h>
#include <stdio.h>
#include <xmmintrin.h>
#include <iphlpapi.h>
#include <Rpc.h>

static void GetMACaddress(void);
static void uuidGetMACaddress(void);

int main(){
    SYSTEM_INFO SysInfo;
    GetSystemInfo(&SysInfo);
    printf("Processors - %d\n" , SysInfo.dwNumberOfProcessors);
    DWORD a , b , c , d , e;
    DWORD BasicLeaves;
    char* VendorID = (char*)malloc(20);
    char* message = (char*)malloc(20);
    _asm {
        pusha
        pushfd
        pop eax
        push eax
        xor eax , 0x00200000
        push eax
        popfd
        pushfd
        pop ecx
        pop eax
        xor eax , ecx
        mov [a] , eax
        }
    if(a & 0x00200000){
        printf("CPUID opcode supported.\n");
        } else {
        printf("CPUID opcode not supported, exiting...\n");
        return 0;
        }

    //DWORD* pa = &a[0];
    //DWORD* pb = &a[1];
    //DWORD* pc = &a[2];
    //DWORD* pd = &a[3];
    //a[4] = 0;
    e = 0;
    __asm {
        mov eax , 0
        cpuid
        mov [BasicLeaves] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        }
    memcpy(&VendorID[0] , &b , 4);
    memcpy(&VendorID[4] , &d , 4);
    memcpy(&VendorID[8] , &c , 4);
    VendorID[12] = 0;

    printf("%d Basic Leaves\nVendorID - %s\n" , BasicLeaves , VendorID);

    __asm {
        mov eax , 1
        cpuid
        mov [a] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        }
    if(d & 0x00000001) printf("FPU\n");
    if(d & 0x00000200) printf("APIC On-Chip\n");
    if(d & 0x00040000) printf("Processor Serial Number Present\n");
    if(d & 0x00800000) printf("MMX\n");
    if(d & 0x01000000) printf("SSE\n");
    if(d & 0x02000000) printf("SSE2\n");
    if(d & 0x08000000) printf("Hyperthreading (HTT)\n");

    if(c & 0x00000001) printf("SSE3\n");
    if(c & 0x00000200) printf("SSSE3\n");
    if(c & 0x00080000) printf("SSE4.1\n");
    if(c & 0x00100000) printf("SSE4.2\n");
    if(c & 0x02000000) printf("AES\n");

    __asm {
        mov eax , 0x80000000
        cpuid
        and eax , 0x7fffffff;
        mov [a] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        }

    printf("%d Extended Leaves\n" , a);

    printf("Processor Brand String - ");
    __asm {
        mov eax , 0x80000002
        cpuid
        mov [a] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        }
    memcpy(&message[0] , &a , 4);
    memcpy(&message[4] , &b , 4);
    memcpy(&message[8] , &c , 4);
    memcpy(&message[12] , &d , 4);
    message[16] = 0;
    printf("%s" , message);

    __asm {
        mov eax , 0x80000003
        cpuid
        mov [a] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        }

    memcpy(&message[0] , &a , 4);
    memcpy(&message[4] , &b , 4);
    memcpy(&message[8] , &c , 4);
    memcpy(&message[12] , &d , 4);
    message[16] = 0;
    printf("%s" , message);

    __asm {
        mov eax , 0x80000004
        cpuid
        mov [a] , eax;
        mov [b] , ebx;
        mov [c] , ecx;
        mov [d] , edx;
        popa
        }
    memcpy(&message[0] , &a , 4);
    memcpy(&message[4] , &b , 4);
    memcpy(&message[8] , &c , 4);
    memcpy(&message[12] , &d , 4);
    message[16] = 0;
    printf("%s\n" , message);

    char VolumeName[256]; DWORD VolumeSerialNumber; DWORD MaxComponentLength; DWORD FileSystemFlags; char FileSystemNameBuffer[256];
    GetVolumeInformationA("c:\\" , VolumeName , 256 , &VolumeSerialNumber , &MaxComponentLength , &FileSystemFlags , (LPSTR)&FileSystemNameBuffer , 256);
    printf("Serialnumber - %X\n" , VolumeSerialNumber);

    GetMACaddress();
    uuidGetMACaddress();

    return 0;
    }

// Fetches the MAC address and prints it
static void GetMACaddress(void){
    IP_ADAPTER_INFO AdapterInfo[16];        // Allocate information
                                            // for up to 16 NICs
    DWORD dwBufLen = sizeof(AdapterInfo);   // Save memory size of buffer

    DWORD dwStatus = GetAdaptersInfo(       // Call GetAdapterInfo
    AdapterInfo,                            // [out] buffer to receive data
    &dwBufLen);                             // [in] size of receive data buffer
    //assert(dwStatus == ERROR_SUCCESS);    // Verify return value is
                                            // valid, no buffer overflow

    PIP_ADAPTER_INFO pAdapterInfo = AdapterInfo; // Contains pointer to
                                            // current adapter info
    do {
        printf("Adapter MAC Address - %X-%X-%X-%X-%X-%X\n" , pAdapterInfo->Address[0] , pAdapterInfo->Address[1] , pAdapterInfo->Address[2] , pAdapterInfo->Address[3] , pAdapterInfo->Address[4] , pAdapterInfo->Address[5]);
        printf("Adapter IP Address  - %s\n" , pAdapterInfo->CurrentIpAddress);
        printf("Adapter Type        - %d\n" , pAdapterInfo->Type);
        printf("Adapter Name        - %s\n" , pAdapterInfo->AdapterName);
        printf("Adapter Description - %s\n" , pAdapterInfo->Description);
        uuidGetMACaddress();

        printf("\n");
        //PrintMACaddress(pAdapterInfo->Address); // Print MAC address
        pAdapterInfo = pAdapterInfo->Next;      // Progress through
                                                // linked list
        } while(pAdapterInfo);                  // Terminate if last adapter
    }

// Fetches the MAC address and prints it

static void uuidGetMACaddress(void)
{
  unsigned char MACData[6];

  UUID uuid;
  UuidCreateSequential( &uuid );    // Ask OS to create UUID

  for (int i=2; i<8; i++)  // Bytes 2 through 7 inclusive
                           // are MAC address
    MACData[i - 2] = uuid.Data4[i];

  printf("UUID MAC Address - %X-%X-%X-%X-%X-%X\n" , MACData[0] , MACData[1] , MACData[2] , MACData[3] , MACData[4] , MACData[5]);
}//*/

Comment: You don't need `pusha`.  Compilers that support that inline asm syntax know what registers you touch.  And you aren't using `popa` until a later asm block, so if you were worried about stepping on the compiler's registers, you're mostly making it worse, not better.  (Unless MSVC doesn't know that CPUID writes EBX?)  Anyway, after checking if CPUID is supported at all, you should normally use a `__cpuid()` intrinsic function instead of inline asm.
Answer: __get_cpuid (unsigned int __level, unsigned int *__eax, unsigned int *__ebx, unsigned int *__ecx, unsigned int *__edx);

Header: #include <cpuid.h>

Note: The processor serial number was introduced on Intel Pentium III, but due to privacy concerns, this feature is no longer implemented on later models.
Source : wikipedia
Answer: Some more details please: operating system, language.
For example on Windows you can get it by using WMI and reading Win32_Processor.ProcessorId.
Comment: Yes, and ProcessorId is some kind of (semi) unique identifier for the processor.
Answer: In windows, I am sure there is a system call, In linux one could try "sudo lshw" but most kernels do not seem to support CPU serial numbers, and preliminary research seems to indicate that the general outrage against uniquely identifiable computers means that there is no perfect answer.
What are you trying to do? Almost certainly someone has done it before and it may be wise to reuse or emulate what they have done.
Answer: You can use CPUID command.
Answer: Use the CPUZ tool: http://www.cpuid.com/cpuz.php
Comment: Since this is a programming site, i would suspect he is asking about how to programmatically get the serial number. He used licensing tag which suspects that he is going to use it to create serial-numbers for his application.
Comment: @Bruce even then, this tool doesn't return serial numbers…
Answer: I guess quite a few compiler do offer some wrapper or the like around the mentioned
command. Here's an example
#include <stdlib.h>
#include <string.h>
#include <intrinsics.h>

_CPUID cpuinfo;
int main(void) {
_cpuid(&cpuinfo);
printf("Vendor: %s\n", cpuinfo.Vendor);
return 0;
}

Output:
Vendor: GenuineIntel

Comment: It was an example of an intrincis, which e.g lcc-win32 offers.
It's not standard. And you have to linke against the proper libraries.
Answer: My suggestion is to get HDD/SSD serial number, many languages have support for this.

CPU serial number


Conflict of Windows Drive Mapping in AD vs from Desktop

Question: With Windows Server 2003, and XP clients what happens if the same network drive letter is mapped via 'Reconnect at login' from the desktop and the homeDirectory AD attribute?
Answer: The AD attribute will take precedence over the local setting.  In other words, whatever path you set to mount via "Reconnect at login" will not be mounted, but the "Home Directory" path in AD will be.


UICollectionView reorder cells after scrolling

Question: I have added this UICollectionViewDelegate with my implementation:
- (UICollectionViewCell *)collectionView:(UICollectionView *)collectionView cellForItemAtIndexPath:(NSIndexPath *)indexPath
{
    SAPCollectionViewCell *collectionViewCell = (SAPCollectionViewCell *)[collectionView dequeueReusableCellWithReuseIdentifier:@"SAPCollectionViewCell" forIndexPath:indexPath];

    NSInteger index = indexPath.item + indexPath.section * 2;

    NSString *imagePath = [_images objectAtIndex:index];

    collectionViewCell.index = index;

    [collectionViewCell setImageWithPath:imagePath];

    collectionViewCell.delegateCell = self;

    return collectionViewCell;
}

In my custom cell I have this method:
- (void)setImageWithPath:(NSString *)imagePath
{
    if (!self.imageView)
    {
        CGRect rect = self.contentView.frame;

        self.imageView = [[UIImageView alloc] initWithFrame:rect];

        [self.contentView addSubview:self.imageView];

        dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{

            UIImage *image = [UIImage imageFromPath:imagePath];

            UIImage *resizedImage = [UIImage imageWithImage:image scaledToWidth:rect.size.width];

            dispatch_async(dispatch_get_main_queue(), ^{

                [self.imageView setImage:resizedImage];

            });
        });
    }
}

So as you can see if the cell does not have UIImageView then I start to create it and in background I get image from local path and resize it to cell content view width, then in main queue I set image to UIImageView.
This part work good but when I try to scroll my UICollectionView e.g. with 10 images I noticed that for example if first 2 top images were disappeared when I scroll down and then when I scroll back to the top, the images are changed placed.
This is state of images before scrolling down:

And this state after I scroll UIColectionView back to the top:

So as you can see first to item changed theirs location. As I have set if (!self.imageView) in my code above it should means that the image never will be created twice or more times.
I opened debugger and was checking this method:
- (UICollectionViewCell *)collectionView:(UICollectionView *)collectionView cellForItemAtIndexPath:(NSIndexPath *)indexPath

So the images paths returned one by one as it worked when UICollectionView just created cells at first time.
At first time first UICollectionViewCell has 0x8fbc1d0
and second has 0x8d0a4c0 address
But when I scroll down and then scroll up the debugger shown me second address at first 0x8d0a4c0 and then shown me 0x8fbc1d0
But I can't understand why items change theirs order. Or maybe here is another issue?
Also I don't have any method in the code that make for example for me reorder for cell. just few UICollection view methods delegate that configure count of cells and create them.
Also if I remove if (!self.imageView) seems everything works good, but then my code every time invoke dispatch that's no good for me, because when I resize image to the cell size I don't need do it twice.
Answer: When you scroll the two items off screen, your CollectionView puts the associated cells back onto the 'queue'.  When you scroll back up again, the dequeueResuableCellwithReuseIdentifier in cellForItemAtIndexPath retrieves them from that queue, but (as you've seen) not necessarily in the same order.  Thus the cell which was item 0 has been used for item 1 and vice versa.  As you've seen, the if (!self.imageView) actually stops the correct image from being loaded.  If you want to avoid retrieving and resizing them each time, then I would store the (resized) imageViews in an array.
Answer: - (void)setImageWithPath:(NSString *)imagePath
{
    if (!self.imageView)
    {
        CGRect rect = self.contentView.frame;

        self.imageView = [[UIImageView alloc] initWithFrame:rect];

        [self.contentView addSubview:self.imageView];

    }

    if (![self.imagePath isEqualsToString:imagePath] && self.imageView.image == nil)
        {
            self.imagePath = imagePath;

            dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{

                UIImage *image = [UIImage imageFromPath:imagePath];

                UIImage *resizedImage = [UIImage imageWithImage:image scaledToWidth:rect.size.width];

                dispatch_async(dispatch_get_main_queue(), ^{
                    if ([self.imagePath isEqualsToString:imagePath])
                    {
                        [self.imageView setImage:resizedImage];
                    }

                });

            });
        }
}


Salesforce Continuous Integration : Only components in the same namespace as the organization can be created through the API

Question: I have a Salesforce CI newbie issue. I'm trying to build a CI pipeline for Salesforce using Bitbucket / Jenkins and the Salesforce Migration tool.  I'm trying to prototype the pipeline against a dev org using a source dump from sandbox which contains a number of licensed packages ( many of which are integrated into standard objects such as Account ). I'm encountering ( understandably enough ) the following error -
"Only components in the same namespace as the organization can be created through the API"
Is there a best practice to deal with issues such as this ? Do I need to download and deploy all licensed managed packages ?
Answer: You have to either remove the managed package items, or install the managed packages. You can use InstalledPackage to help automate the process if you want to go down the path of installing dependencies, or you can install them manually; Developer Edition orgs, like Sandbox orgs, can install any package and get an unlimited Site license for evaluation purposes. If installing is not suitable for your purposes, you'll want to come up with a way to strip any references to those packages from your code and metadata, which may be easier said than done.
Comment: I'll try your suggestion re. InstalledPackage. We really do need this to be fully automated. Thanks !
Answer: I think your problem is that when you're retrieving from the sandbox you're retrieving using a non-specific package.xml file.  As a result, when you retrieve the Account object, you get everything on Account including custom fields with namespaces from those installed packages.  When you take that metadata and deploy to another org, it's trying to create fields that are in a namespace you don't own hence the error.  You can get around this in one of two ways:

Use an Unmanaged Package in your Sandbox org to select the metadata you want to retrieve.  Go to Setup -> Create -> Packages and create a package.  You won't ever distribute this package from here, but a package is a useful container that you can add your custom metadata into.  Once you've added the metadata you want into the package, use the packageNames argument to the sf:retrieve call in your Ant targets.  Set the value of packageNames to the package you created in the org.  With this retrieve mode, you don't provide a package.xml file.  The retrieve returns only the metadata in the package.
Manually create a specific package.xml file for what you want.  For example, don't retrieve the whole Account CustomObject in your package.xml file.  Instead, just retrieve the CustomFields you want.  I'd recommend option 1 as this can get painful to maintain.

If your custom metadata references fields or other metadata from any of the installed packages, you'll still need to use InstalledPackage to automate the installation in your build scripts.
FWIW, I spent the last 3 years building out our CI process around Ant + Force.com Migration tool, Jenkins and Github.  From my experience, Ant can become a burden with more complex build automation logic like handling dependencies.  It's possible and I built CumulusCI around Ant which I've been running for the last 3 years.  That inspired me to spend the last 6 months rewriting it all in Python as CumulusCI 2.0 version that's in alpha now.  Feel free to poke around for ideas for your scripts.
Comment: Thanks for the feedback; however I realized that wasn't my problem. I was able to resolve it by using installedPackage as suggested by sdfcfox. I'm facing a different issue while trying to install ( using the api ) npsp 3.0. This is what I see .
Comment: All Component Failures:
1.  installedPackages/npsp.installedPackage -- Error: (Contacts & Organizations 3.4) A required package is missing, Details: Package "Contacts & Organizations", Version 3.4 or later must be install\
ed first.
Answer: I recently had this issue. The issue occured because of different format/type of component.
When we're deploying using MDAPI(from SFDX Command/Ant/Workbench etc.) it requires different format and with the advent of scratch org(or SFDX) folder structure, it uses different formatting of files(specially object files).
To resolve the issue - I used sfdx force:source:convert command to convert Scratch org compatible folder structure into MDAPI compatible folder structure.
Hence the issue resolved.


Viewing disk usage of temporary buffers of running queries in Postgres

Question: I know how to get the sizes of relations or total table disk usage per table in Postgres 9.5 (https://wiki.postgresql.org/wiki/Disk_Usage), but I have some queries running whose temporary disk buffers use significant amounts of space. How can I see how much each is using? My disk is slowly running out of space due to many long-running queries, and I need to kill one of them.
Answer: If you have queries generating temporary files, it could indicate that work_mem is not sufficiently sized or that the query is not optimised.
Try setting the parameter log_temp_files = 0 in your postgresql.conf.  This will log the creation of temporary files and the statement being executed.
Additionally an explain analyze of your query may show disk sort happening.
Taking the log output, you will see the size of temp  files created by your query.  You can use this to either fine tune the query or fine tune the work_mem parameter.
Comment: It says things like `2016-06-02 11:24:43 PDT LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp20410.11", size 174272980`. I'm assuming the size is in bytes and that it's creating a lot of ~200MiB temp files. I expect my queries to be using lots of temporary files because my tables are around 200GiB, and I've tuned Postgres for my machine with 64GiB of RAM.


Jquery form validation not working as wished? Must be a better method?

Question: Take a look at the following link, you will see I have a simple form with various fields that are required. I have put together a small script to validate entry before submission. Normally the random numbers are generated using php but for the example I have just added them manually.
The solution does seem to work but I would prefer as one input is completed the error would disappear. Currently this works for the standard inputs that just require some kind of entry  but doesn't work when an input requires a valid email or sum. If a correct email is entered, but the verify question hasn't been entered the error still appears on the email field.
http://jsfiddle.net/marcsanders/xgZeV/
Any help much appreciated!
Answer: Try the following:
$('input.required').change(function() {
 $(this).parent('div').removeClass('error');
});

Updated jsFiddle here.
You may also consider using a html5 required attribute on elements that are required, rather than using javascript.
Answer: You can try adding "keydown" events on all mandatory inputs and remove the "error" class if at all it is there attached to the input.


Receiving ERC20 tokens as paymant inside my smart contracts

Question: Is there a way to receive X amount of ERC20 tokens to transact certain methods inside my smart contracts?
So let's say a mint method is payable and receives ether as a value instead, it requires the msg.sender to send X amount of ERC20 tokens to the smart contract in order to perform the transaction.
Answer: A token is nohting more than a variable (a mapping from an address to a uint256) in the smart contract, so in principle you could code something like this.
<code>   function Whatever() public returns (something) {
       balanceOf[msg.sender] -= (some thokens)
       ... execute the rest of the function
   }
</code>
But you should be aware of taking care of the total amount of tokens of the contract. If the msg.sender is "spending" some tokens, it must be transfered to someone else. So the better solutions is to call the tranfer function inside the Whatever function
<code>   function Whatever() public returns (...) {
        Transfer(msg.sender, some addreess, amount)
        ... the rest of the code
    }
</code>
You just need to think the logic of your contract, the case is that tokens are a variable that can be manipulated by the contract the way you want.


Polygon trigger colliders for 2D platformer

Question: I'm creating a 2D platformer in MonoGame and I'm thinking of using AABB as physical colliders for characters and then have a trigger polygon collider on each character for checking for example if an enemy's weapon is touching the player, making it more precise. The polygon collider is animated so every frame in every animation has its own points. That's a lot of work. Is that precision needed? What is usually done in this situation?
Answer: Updating a primitive collider position is already costly, polygon colliders are really expensive by definitnion, i only use it for static objects, changing a polygon collider position is really expensive. And on top ot that changing a polygon collider actual polygon.?
Its probably not a good idea, you could try but i dont thinl you will succeed. Most people use primitive colliders next to each other to fake a more complex geometry.
Comment: So, should I have an AABB for physical collisions and then some primitive trigger colliders for character collision testing that get updated depending on the animation and frame of the animation?
Comment: Yes, i wouod do that!
Comment: If you decide to try the polygon collider editing in runtime solution, let me know your results!
Comment: I don't think the performance results will be visible with just a handful of characters, but if you have tons of them in the same level the framerate might drop, but I don't know, maybe my PC is too good to be slowed down by a 2D platformer under (almost) any circumstance. The performance of a polygon collider depends on the amount of points and I try to keep it low (5-8).
Comment: But maybe the collision is not really accurate. Anyways, i wanna know it :D


Can you still become Arch-Mage of the Mage's Guild if you kill Trebonius before getting the quest to become Arch-Mage?

Question: Obviously, I would taunt him into attacking me, or use a Frenzy spell on him so I can kill him without being permanently expelled, but will the quest work properly if he's already dead, rendering me unable to give him the letter to become Arch-Mage peacefully, or to challenge him in the arena in Vivec?
Answer: Without the use of console commands? You can effectively do anything with console commands.
The only way other than that, if you've already killed him, is if you killed him while already being a "Master Wizard" at the actual time you killed him. (The rank in the guild directly preceding Arch-Mage)
If you weren't, then no, you can't become Arch-Mage.
If you are not opposed to using Console Commands, the easiest thing to do would be

Find his corpse.
Open console commands (~ key on PC, typically)
Click on his corpse.
Type 'resurrect' without the apostrophes.
Press Enter. He should then be alive again. If he attempts to attack you afterwards, open console commands again, referencing him, and enter 'stopcombat'. Then you can bribe him to like you more, and not attack you.

Hope this helps. :)
UPDATE
If you can't find his body, you can also try:
[resurrect trebonius artorius] (without the brackets)
or
["trebonius artorius"->resurrect] (with quotes, no brackets though)
If he still wants to kill you when he sees you,
["trebonius artorius"->setfight 0] (with quotes, no brackets)
If you want to see him immediately after entering this, you can use the placeatme command:
[placeatpc "Trebonius" 20,10,1]
LASTLY
If you want to be cheap about it, and skip straight to completing the guild quest and becoming Archmage, you can
[PCRaiseRank "Mages Guild"] in order to gain a rank
[Journal MG_Guildmaster 100] to complete the quest.


How to publish single sensor_msgs/LaserScan over ROS from two sick_tim sensor

Question:
How can I publish single sensor_msgs/LaserScan over ROS from  two sick_tim sensor.

Originally posted by naka on ROS Answers with karma: 18 on 2015-03-04
Post score: 0
Answer:
You can't, and you probably don't need to.
The ROS LaserScan message represents a scan in polar coordinates - as an angle and an offset from a center point. Since your laser scanners don't have the same center point, it doesn't make sense to put data from both lasers into the same message.
If you describe your robot and your application in more detail, we can probably help you find a better way to do what you're trying to do. (For example, the navigation stack supports multiple lasers).

Originally posted by ahendrix with karma: 47576 on 2015-03-04
This answer was NOT ACCEPTED on the original site
Post score: 1

Original comments
Comment by naka on 2015-03-04:
I want to move my robot with the navigation stack!
The sick_tim sensors are installed in front and back of the robot.
Comment by ahendrix on 2015-03-05:
If you're just trying to use your lasers with the navigation stack, you should use two separate topics and configure two observation sources for navigation.
Answer:
As mentioned by Lorenz on this thread:
http://answers.ros.org/question/37794/using-multiple-laserscan/
It seems that you can simply publish on the same topic from multiple laser scan sources, since each laser scan message includes frame and time information to differentiate. For this you have to set up your URDF correctly.

Originally posted by Valts with karma: 41 on 2015-03-04
This answer was ACCEPTED on the original site
Post score: 0


Hair Comb not working

Question: I'am trying to make a rug with all the simple particle rug settings. I want to comb the hair with particle edit but whenever i enter particle edit mode, my hair settings all distort up.
This is what it looks like in object Mode

Right when I enter edit particle mode, this happens

Getting out of particle edit doesnt fix this and i have to undo everything. I only need to comb the hair. Please help.
Comment: I'm probably wrong but try applying scale, and change emit from setting. Does your plane have edges where bald spots are created?
Comment: Applying scale doesn't work either. No there arent bald edges
Answer: Delete any custom normals or Vertex Groups on your object properties. Delete any other irrelevant information (this is caused by a Smoothing Group from other software).
Also check you don´t have unchecked the eye icon and the edit icon on one of your modifiers. This is the reason why you see something in Edit Mode and something else in Object Mode.
Last but not least, you have to press Ctrl +  A on your rug to apply transformations.

Hair Comb not working


Is there any precautions I should take when using an ssd instead of an hdd?

Question: The only thing I'm aware of so far is that you shouldn't defrag a solid state drive or that ruins the life expectancy of it. Is there anything else?
What about 3rd party programs like ccleaner. I have ccleaner set to clean every time I turn my computer on. Is that bad for a ssd?
Comment: It's bad for the operating system)
Comment: What's bad for the operating system?
Running ccleaner?
Comment: The short answer - yes. Well, or try to talk in detail, making Ccleaner.
Comment: How so?
All it does is clear temp files
Comment: Not worth all the temporary files to delete, especially during OS. With the registry situation is more complicated, few can do something useful there. On the basis of their experience in different companies, I can say that at this point leads to errors CCleaner.
Answer: There has been a lot of disambiguation about running things like CCleaner and Defrag on an SSD. Running constant sequential writes can in theory shorten the life expectancy of an SSD. However you're only loosing days off of an life expectancy of years. The drive will outlast your use of it for sure. The thing about Defrag is it won't speed up your drive in the slightest since SSD's can read so quickly that fragmented files are no longer the issue they used to be. CCleaner will not hurt your drive in the slightest. Here's a great link for a thread from Tom's that talks about defraging an SSD.
In short I went to an SSD raid 0 in my rig two years ago and would never go back
Ben
Comment: Install Windows fresh, as it uses different settings, to adapt to C: being SSD drive.  The fears are overdone, having win page file on SSD is great.  If you have small SSD moving things like Browser cache/Temp files and Downloads could be worth doing, to reduce writes but for a larger one it's not worth the effort as you'll get nowhere near using it's Write life expectancy, before it's totally obsolete.  Looking at TR's SSD Endurance Test I wouldn't worry http://techreport.com/search_google.x?cx=013722762777297270298%3A8y3qir4a7d4&ie=UTF-8&q=SSD+Endurance+Experiment&sa=
Answer: One precaution you should take is to backup as normal Alex.
That is not to say SSDs are unreliable. Far from it. I've personally deployed and supported lots of PCs and laptops for many years and I have to say that SSDs are a joy to work with in comparison to the traditional spinning hard disks. I've yet to see an SSD fail.
Speaking to several laptop engineers however, when SSDs fail they can fail suddenly without any warning. This is unlike with the traditional hard disks where you can hear them start to make that whining scratching noise when they start to degrade, thus you have time to make a backup. So back up as normal Alex.
Hope that helps.
Comment: Making System Image of SSD regularly is good precaution.  I have had SSD problems in past (was an early adopter) but firmware got sorted out so not seen for long time.
Answer: If you care about your SSD life time then you need to be aware that due to limited write/erase cycles number per sector the device may start dying after a few years of serious usage. From this perspective it is recommended to put the most rewritable data on the HDD. OS will run much faster being placed on SSD, of course, but OS itself has files that are rewritten often.
In Unix-like OS you can afford you to keep your home directory at any different partition. Myself have it with SSD + HDD couple. In Windows, I don't know how separate OS itself and Users or Documents and settings folders from each other.
As I said if someone cares about lifespan. Of course to get all speed purchased most writing should be on SSD.
At the time I am writing this I was also told, that

The data that's being rewritten most often is precisely what you want on the SSD, as it's where you're going to get the most speed improvement from the system. Buying and SSD and then only using it for infrequently used files makes no sense to me. Yes this means it will wear out faster, but it also means you get more speedup per dollar spent. Thanks for this to James Polley.
Modern SSDs have write lifetimes of 17.5 GB to 70 GB per day, every day to give it a 3 year lifespan. If you are using a modern drive the IO activity of your OS reading and writing to stuff like the page file will barely scratch it. What you said about the OS wearing it out was true for the first generation of SSDs but is no longer true with modern ones. Thanks for this to Scott Chamberlain.


What is the magnetic field due to an electromagnet at a point that is not along its axis?

Question: For e.g. at a point below(or perpendicular) to a cuboidal/cylindrical core electromagnet?
I am confused whether the magnetic field for an iron-core electromagnet is limited to the axis passing through it because if that is the case how can it be used for the various applications that requires applying force on other metals?
Please specify the method to calculate the field,if any.
I am a beginner so kindly pardon any mistakes.
Comment: What you're looking for is an introduction to Maxwell's Equations (assuming you already know calculus). http://electronics.stackexchange.com/a/20010/35022
Answer: A magnet can be treated as a magnetic dipole and the magnetic field can be calculated in similar form to the calculations used for an electric dipole. By convention, the vector is taken to point away from the magnet at the N pole and toward the magnet at the S pole. If you imagine the magnet in the center of a clock face, with N pointing to 12, the vectors rotate through \$360^\circ\$ as you move around from 12 to 6 on the clock and diminish to a minimum of strength at 3 and 9 on the clock, where the magnitude will be half what it was at the same distance on the clock face at 12 and 6. The field will vary in strength \$\propto \tfrac{1}{r^3}\$.
If you want to measure the actual field, you might first figure out the strength of the Earth's magnetic field (tangent to the surface) at your location. In the US, it's generally about \$2\times 10^{-5}\$ Tesla, though in Maine it's about 25% weaker and in Florida it's about 50% stronger. You can then take a sensitive compass and make some measurements outside, away from any metal structures (or metal table materials.) Set the compass so that it is undisturbed and settles down to "pointing north." Now, approach the compass with your magnet, where you approach along a line that is perpendicular to the N-S line the compass is currently describing, holding your magnet with N (or S) pointing straight at the center of the compass. When the compass needle rotates exactly \$70^\circ\$ away from its original direction (and towards or away from) your magnet, measure out the distance from the center of the compass to the center of your magnet. At this point:
$$\vec{B}_{magnet} = \vec{B}_{earth}\cdot \textrm{tan}\left(70^\circ\right)$$
If you are in the US where \$\vec{B}_{earth}\approx 2\times 10^{-5}\:\textrm{T}\$, then you have now determined the distance where \$\vec{B}_{magnet}\approx 5.5\times 10^{-5}\:\textrm{T}\$. Using this distance, you can now compute the magnetic dipole moment of your magnet as:
$$\mu = \frac{4\: \pi\cdot B_{magnet}\cdot r^3}{2\:\mu_0}$$
If you stay in SI units, the answer results in magnetic dipole moment units of \$\textrm{A}\cdot \textrm{m}^2\$ (or, equivalently, Joules per Tesla.)
You can do these experiments with various magnets and/or electromagnets, if you prefer. Once you know it's magnetic dipole moment, you can sit down and map out the field on paper, if you like.
If you are in a different place in the world, you should be able to find a reasonable value to use for \$\vec{B}_{earth}\$ near your location. Just follow the above equations using your own value, instead.
A common bar magnet weighing a couple of ounces might yield \$3\:\textrm{A}\cdot \textrm{m}^2\$, for example.
Answer: For analyzing the magnetic field from a coil at some distance, the fall back position is the biot savart law. Here's the simple example alluded to in the question: -

And, for the magnitude of the fields off-centre I've found that the only way to practically do this is go back to the basic biot savart law: -


fdisk -l shows 16 ram disks /dev/ram0 ... /ram15

Question: Ever since I upgraded to 15.10, fdisk -l reports 16 ram disks (/dev/ram0 ... /dev/ram15).
I'm a bit unsure what those are needed for. Is it safe to delete them? If not, how can I get rid of that fdisk output?
Answer: This is perfectly normal on Linux systems. It's kind of a preparatory action for the case that the RAM disks should be required. Each of them has a size of 64 MiB, a very low value. If necessary, the size will be increased automatically.
Why suddenly 16 RAM disks are available in Wily, can be explained only with difficulty.
I have tested the default RAM disks on:

CentOS 7 – No RAM disks
Fedora 23 – No RAM disks
Ubuntu 14.04 – No RAM disks
Raspbian Jessie – 16 RAM disks (4MiB)

Source

The RAM disk driver is a way to use main system memory as a block device.  It
  is required for initrd, an initial filesystem used if you need to load modules
  in order to access the root filesystem (see Documentation/initrd.txt).  It can
  also be used for a temporary filesystem for crypto work, since the contents
  are erased on reboot.
The RAM disk dynamically grows as more space is required. It does this by using
  RAM from the buffer cache. The driver marks the buffers it is using as dirty
  so that the VM subsystem does not try to reclaim them later.
The RAM disk supports up to 16 RAM disks by default, and can be reconfigured
  to support an unlimited number of RAM disks (at your own risk).  Just change
  the configuration symbol BLK_DEV_RAM_COUNT in the Block drivers config menu
  and (re)build the kernel.
Comment: Note that they do not use up any memory if you never write anything to them.  What seems to have changed is that the kernel did not used to list ramdisks in /proc/partitions, but now does, so fdisk -l reports on them.
Comment: Thanks for reminding me. The answers were enlightening, explaining the situation, so thanks for those. Unfortunately, I still don't have a clue how to suppress that messy disturbing output.
Comment: And this changed from 15.04 to 15.10?
Comment: @RudiC: As you're a reputation 6 user: If this answer helped you, don't forget to click the grey **☑** at the left of this text, which means [Yes, this answer is valid](http://askubuntu.com/help/accepted-answer)!  **;-)**
Comment: Just FYI - checked vanilla Debian Jessie, and it's the same result you got for Raspian Jessie.
Answer: No idea why fdisk is suddenly reporting /dev/ram.
You can however tell fdisk to only report specific devices.
fdisk -l /dev/sd*

Will list real drives.
Alternatively you could also use parted and lsblk.
Parted output for one drive here.
Model: ATA Samsung SSD 840 (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type      File system  Flags
 1      2096kB  120GB   120GB   extended               boot
 7      2097kB  26.2GB  26.2GB  logical   ext4
 5      26.2GB  36.7GB  10.5GB  logical   ext4
 6      36.7GB  47.2GB  10.5GB  logical   ext4

Corresponding lsblk output
    NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 111.8G  0 disk
├─sda1   8:1    0     1K  0 part
├─sda5   8:5    0   9.8G  0 part /mnt/Links
├─sda6   8:6    0   9.8G  0 part
└─sda7   8:7    0  24.4G  0 part /

Comment: note that on some boxes (ACPI versions?) physical devices are ***h***d*, not ***s***d*.
Comment: And NVMe devices are /dev/nvme*, and MMC devices (some Chromebooks use these) are /dev/mmcblk*
Answer: I know this thread is old, but I came across it only recently.
After installing Slackware 14.2 I got the same 16 RAM disks in the output of
fdisk -l. I investigated a little further and found that in the 'util-linux'
package, which fdisk (among others) is part of, the selcetion of what fdisk considers as block device changed substantially. In the util-linux package version 2.21 this decision is based on the reported disk geometry while in the current version 2.72 the output of /proc/partitions is parsed.
According to my searches on the internet the ramdisks have been there in Linux since kernel 2.4, fdisk did just not show them. Since I am annoyed by the listing of many "disks", which are no real disks, I made a patch for fdisk:
    diff -Nur util-linux-2.27.1_ori/disk-utils/fdisk-list.c util-linux-2.27.1_fdisk-no-ram-disks/disk-utils/fdisk-list.c
--- util-linux-2.27.1_ori/disk-utils/fdisk-list.c   2015-10-06 08:59:51.572589724 +0200
+++ util-linux-2.27.1_fdisk-no-ram-disks/disk-utils/fdisk-list.c    2016-08-16 15:55:14.840952091 +0200
@@ -312,6 +312,10 @@
        if (devno <= 0)
            continue;

+       /* dont list RAM disks */
+       if (strstr(line, "ram") && devno >= 256)
+           continue;
+
        if (sysfs_devno_is_lvm_private(devno) ||
            sysfs_devno_is_wholedisk(devno) <= 0)
            continue;

Maybe this helps some others...
Comment: This forces you to recompile `fdisk`
Answer: The post by Johannes is correct.  The ram-disks have been in the kernel for a long time, it is the behavior of fdisk that changed.  Instead of patching fdisk, I wrote a simple perl script (5 lines of code, 6 comment lines)  to handle the issue.  I put it into ~/bin/fdisk-l, and now I just remember not to put a space between fdisk and -l.
#! /usr/bin/perl -w
# Run fdisk -l and filter out the 16 /dev/ram devices.
# Sun Mar 5 16:13:45 2017. Jeff Norden, jeff(at)math.tntech.edu

$_=`sudo fdisk -l`;  #include sudo we don't have to be root

# weed out ram disks. The seemingly contradictory s (single) and m (multiline)
# flags allow "." to match "\n" and "^" to match at all beginning-of-lines.
s|^Disk /dev/ram.*?\n\n\n||smg;

# Do better than blank lines separating devices. Handle odd cases when there
# are more than two blank lines between devices or none at the end.
$hrule= '='x60 . "\n";
s/(\n\n\n+)|(\n+$)/\n$hrule/g;
print($hrule, $_);

As of April 2017, the ram disks no longer appear by default with the current Ubuntu kernel, so this issue  is resolved.  See: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1593293
Answer: This behaviour is governed by kernel options that you can only change by recompiling a custom kernel. You can change the size of the ram* devices using a GRUB parameter ramdisk_size but not the count. This is useless, because even if you have lots of memory every ramdisk will increase to whatever size you set. So for instance if you want an 8GB ramdisk--which I do, see below--you'll get 16x 8GB instances. I don't know whether this is harmless if you don't use most of them, but I'm reluctant to brick my system if it isn't.
I want to use a 8GB /dev/ram device to mirror with an 8GB hard disk partition for the specific purpose of putting a hot disk area on it. My application will automatically write the blocks out to regular storage based on the free space, so it doesn't matter that it's small.
With write-behind under mdadm,this should have the effect of making writes blazing fast if they are bursty, with the HDD side of the mirror catching up when things are quieter to provide at least some data protection. I have used this setup with Solaris, but it doesn't seem to be possible with Linux as it comes out of the box.
Since RAM is orders of magnitude faster than SSD, this should be a win, but I can't try it. As others have noticed, if you build a RAID1 with tmpfs, it won't reassemble at boot because the step that initialises tmpfs is far too late in the boot process--at mountall. Your mds are well and truly built by then, so it fails, and you have to rebuild it manually.
OTOH /dev/ram* devices would be perfect for this--if you could configure them. They are the very first thing that gets set up, and ram0 is the initial / filesystem.
Comment: This is a good answer. Please remove the "enhancement request" however, as it is not appropriate for an answer.
Comment: I'd still like to make the enhancement request. What is the best way?
Answer: I've been bugged by this for a few years now. Jeff Norden made a cool perl script. It lost the colors of fdisk though so I added them and named it lsdisk. I added --color=always and removed the ^ that broke the regex because of the newly displayed color codes:
#! /usr/bin/perl -w
# Run fdisk -l and filter out the 16 /dev/ram devices.
# Sun Mar 5 16:13:45 2017. Jeff Norden, jeff(at)math.tntech.edu

$_=`sudo fdisk -l --color=always`;  #include sudo we don't have to be root

# weed out ram disks. The seemingly contradictory s (single) and m (multiline)
# flags allow "." to match "\n" and "^" to match at all beginning-of-lines.
s|Disk /dev/ram.*?\n\n\n||smg;

# Do better than blank lines separating devices. Handle odd cases when there
# are more than two blank lines between devices or none at the end.
$hrule= '='x60 . "\n";
s/(\n\n\n+)|(\n+$)/\n$hrule/g;
print($hrule, $_);