tcl text processing - rearrange values in rows and columns based on user defined value

Question: I am new to tcl and would like to use it in text processing of a simple case. The following format is in Liberty (.lib file) which is used in chip design. I would be truly indebted for any help on this.
Here is a snippet of my file (text processing to be done only on the "values")
timing () {
    related_pin : "clk";
    timing_type : setup_rising;
    rise_constraint (constraint_template_5X5) {
      index_1 ("0.01, 0.05, 0.12, 0.2, 0.4");
      index_2 ("0.005, 0.025, 0.06, 0.1, 0.3");
      index_3 ("0.084, 0.84, 3.36, 8.4, 13.44") ;
      values ( \
        "1.1, 1.2, 1.3, 1.4, 1.5", \
        "2.1, 2.2, 2.3, 2.4, 2.5", \
        "3.1, 3.2, 3.3, 3.4, 3.5", \
        "4.1, 4.2, 4.3, 4.4, 4.5", \
        "5.1, 5.2, 5.3, 5.4, 5.5", \
        "6.1, 6.2, 6.3, 6.4, 6.5", \
        "7.1 ,7.2, 7.3, 7.4, 7.5", \
        "8.1, 8.2, 8.3, 8.4, 8.5", \
        "9.1, 9.2, 9.3, 9.4, 9.5", \
        "10.1,10.2,10.3,10.4,10.5", \
        "11.1,11.2,11.3,11.4,11.5", \
        "12.1,12.2,12.3,12.4,12.5", \
        "13.1,13.2,13.3,13.4,13.5", \
        "14.1,14.2,14.3,14.4,14.5", \
        "15.1,15.2,15.3,15.4,15.5", \
        "16.1,16.2,16.3,16.4,16.5", \
        "17.1,17.2,17.3,17.4,17.5", \
        "18.1,18.2,18.3,18.4,18.5", \
        "19.1,19.2,19.3,19.4,19.5", \
        "20.1,20.2,20.3,20.4,20.5", \
        "21.1,21.2,21.3,21.4,21.5", \
        "22.1,22.2,22.3,22.4,22.5", \
        "23.1,23.2,23.3,23.4,23.5", \
        "24.1,24.2,24.3,24.4,24.5", \
        "25.1,25.2,25.3,25.4,25.5", \
      );
    }

So all the "values" are in a 25 rows x 5 columns lookup table format which I wish to change to a 5 rows x 5 columns table format. To accomplish this, I would like to ask the user which of the 5 values in index_3 he/she wants before mapping is done as follows (while also removing the index_3 line):
C is the user defined column based on index_3: (column 1 for 0.084, column 2 for 0.84, column 3 for 3.36, column 4 for 8.4, column 5 for 13.44). *Only 1 value can be chosen by user
Mapping scheme:
   1,C -> row 1 column 1
   2,C -> row 2 column 1
   3,C -> row 3 column 1
   4,C -> row 4 column 1
   5,C -> row 5 column 1
   6,C -> row 1 column 2
   7,C -> row 2 column 2
   8,C -> row 3 column 2

etc ..
For EXAMPLE, say a user chooses column 1 (value 0.084 from index_3) --> which 'chooses' entire 1st column of the data in 'values' to be text processed/arranged
Hence, resulting text processing done by tcl based on the mapping scheme should be:
index_1 ("0.01, 0.05, 0.12, 0.2, 0.4");
index_2 ("0.005, 0.025, 0.06, 0.1, 0.3");

 values ( \
        "1.1, 6.1, 11.1, 16.1, 21.1", \
        "2.1, 7.1, 12.1, 17.1, 22.1", \
        "3.1, 8.1, 13.1, 18.1, 23.1", \
        "4.1, 9.1, 14.1, 19.1, 24.1", \
        "5.1, 10.1,15.1, 20.1, 25.1", \
  );

My strategy is this:

search for "rise_constraint" to zero in on which values to text process in the entire file
comment out line "index_3 (...)"; add /* and */ at beginning and end of line in reprinted processed file (optional)
convert 'values' from 25 rows x 5 columns table to 5 rows x 5 columns table BASED on Index_3 value chosen ("user defined column choice")
re-print other lines as is (including the text processed 'values')

I tried my very best to explain my coding request. Can any of you meastros please help me think of a proper way in tcl I can do such text processing? Many thanks!!
Comment: You are going to have to make a start yourself....
Comment: How large is the file (well, order of magnitude)? 1kB? 1MB? 1GB? 1TB?
Comment: You should also edit your previous question before posting a new one if it's a revamp of the previous one.
Answer: The first step is to know how to represent the data in tcl in its final form.
This is just one possible solution.
dict set risedata constraints {
  constraint {
    0.084 { 0 { 0 1.1 1  6.1 2 11.1 3 16.1 4 21.1 }
            1 { 0 2.1 1  7.1 2 12.1 3 17.1 4 22.1 }
            2 { 0 3.1 1  8.1 2 13.1 3 18.1 4 23.1 }
            3 { 0 4.1 1  9.1 2 14.1 3 19.1 4 24.1 }
            4 { 0 5.1 1 10.1 2 15.1 3 20.1 4 25.1 }
    }
  }
  indexes { 1 { 0.01 0.05 0.12 0.2 0.4 }
            2 { 0.005 0.025 0.06 0.1 0.3 }
            3 { 0.084 0.84 3.36 8.4 13.44 } }
}

set c 0.084
puts "$c 2 3: [dict get $risedata constraints constraint $c 2 3]"
puts "idx1 3: [lindex [dict get $risedata constraints indexes 1] 3]"
puts "idx2 3: [lindex [dict get $risedata constraints indexes 2] 3]"

Then knowing where you need to be, loading the .lib is just a straightforward parsing problem:
set fh [open z.lib r]
set inval false
while { [gets $fh line] >= 0 } {
  if { [regexp {\);} $line] } {
    set inval false
  }
  if { [regexp {index_(\d+)} $line all idx] } {
    regsub {^[^"]*"} $line {} d
    regsub {".*} $d {} d
    regsub -all {,} $d {} d
    dict set risedata constraints indexes $idx $d
  }
  if { $inval } {
    regsub {^[^"]*"} $line {} d
    regsub {".*} $d {} d
    regsub -all {[ ,]+} $d { } d
    set row [expr {$rcount % 5}]
    set column [expr {$rcount / 5}]
    set i 0
    foreach {v} [split $d { }] {
      set c [lindex [dict get $risedata constraints indexes 3] $i]
      dict set risedata constraints constraint $c $row $column $v
      incr i
    }
    incr rcount
  }
  if { [regexp {values} $line] } {
    set inval true
    set row 0
    set rcount 0
  }
}
close $fh

puts $risedata
set c 0.084
puts "$c 2 3: [dict get $risedata constraints constraint $c 2 3]"
puts "idx1 3: [lindex [dict get $risedata constraints indexes 1] 3]"
puts "idx2 3: [lindex [dict get $risedata constraints indexes 2] 3]"

Comment: @msb Thanks for the clarification :) I just joined stackoverflow yesterday so I did not know about the green checkmark option (until now that is). I have marked it as correct. My apologies if I misunderstood you. I shall redirect Brad to my follow up link. Good day!
Comment: That was phenomenal!! Clear and concise. Thank you so very much. I ran the script for now and it returns what I exactly asked for! You are a LIFESAVER! I will have to definitely take some time to understand what you rustled up - the 'dict' and 'regexp' commands are new to me. Seems like dict is used in the way you would handle arrays. One tiny question, how do you propose I can overwrite my original.lib file 'values' with the updated 'values' returned by your variable 'constraints' (after the user inputs his index_3 value)? Also, do let me know how I can ref you for my project. Thx
Comment: If you need to create a new .lib file, you will need to upgrade the parser above to handle the rest of the fields.   Once you have all the needed values in the dictionary, writing out a new .lib file is pretty easy.  References: [dict man page](http://www.tcl.tk/man/tcl8.5/TclCmd/dict.htm) [regexp man page](http://www.tcl.tk/man/tcl8.5/TclCmd/regexp.htm) [regsub man page](http://www.tcl.tk/man/tcl8.5/TclCmd/regsub.htm).   For writing out a file, you would use open, close, dict, foreach, puts.
Comment: Thanks a lot for the references and strategy! I've tried to follow your advice but have not a find a way to replace the values with the new values including the ',' and '\' characters and arranging the values in a 5x5 format (as shown in my question). This is only last piece of help I earnestly ask of you. I will reference your name on my project. This has been of immense help!
Comment: I am struggling to remove the indexes between the values in say {0 3.1 1 8.1 2 13.1 3 18.1 4 23.1} and change it to {3.1 8.1 13.1 18.1 23.1} before overwriting this in .lib file. After removing the indexes, I will have to insert the ',' and '\' at the end for each line (iterated through a for loop). You inputs will be extremely appreciated. Sorry to bother you so much but I don't think I've ever come across a tcl maestro like you so far and hence I'll get to learn a lot from your inputs. Thanks!
Comment: @edaloke if the answer solves your problem, you can accept it as the correct answer, at least as a courtesy to the answerer if nothing else... ;)
Comment: @msb The answer is definitely correct and I have courteously acknowledged it (check comments please). I am following up on a continuation of the problem. I'm just requesting for further inputs from Brad if he has the time. Behooves me to learn from the best.
Comment: @edaloke I'm talking about marking it as correct, it's the green checkmark below the votecount. :) Don't worry, you can keep discussing after you accept it, or you can create another question and direct him there, if it's reasonably different from your original question. I'm not saying you're doing it wrong, just explaining the common use of stackoverflow from what I know. (Y)
Comment: @edaloke welcome to SO. :) I saw you were new, that's why I decided to clarify. Not everyone is on this site for the points, but marking as correct is usually seen as a combination of showing gratitude and awarding points all at once. ;) Good day!
Comment: You can put multiple variable names in a foreach loop: `foreach {k v} [dict get $risedata constraints constraint 0.084 0] {
  puts -nonewline "$v, "
}
puts " \\"`
Just wrap this in another loop to increment the main index from 0 to 5.
Answer: I finally got really interested in this, and against my better judgement coded up a complete answer. I'm not going to document it in any way. Read it, read the docs for commands you don't understand, then come and ask questions.
Looking at your liberty file, I see it's very close to native Tcl syntax. So you can create a few procedures named "timing", "rise_constraint", etc, and you can basically run it as a script.
package require struct::list

######################################################################
proc main {libfile} {
    global lines idx3 vals
    set lines [list]
    set idx3 [list]
    set vals [list]

    evaluate_liberty $libfile

    set idx [get_choice "select an index_3 value: " $idx3]
    set column [struct::list mapfor elem $vals {lindex $elem $idx}]

    set newvalues [list]
    for {set i 0} {$i < 5} {incr i} {
        lappend newvalues [lrange $column [expr {5*$i}] [expr {5*($i+1)-1}]]
    }

    print_liberty $newvalues
}

######################################################################
proc evaluate_liberty {libfile} {
    set fh [open $libfile r]
    # handle known syntax error in liberty file
    set contents [string map {\", \"} [read -nonewline $fh]]
    regsub -all -line {\s+$} $contents {} contents
    close $fh

    uplevel #0 $contents
}

proc get_choice {prompt values} {
    while {1} {
        for {set i 0} {$i < [llength $values]} {incr i} {
            puts stderr [format "%2d. %s" $i [lindex $values $i]]
        }
        puts -nonewline stderr $prompt
        gets stdin answer
        if {[string is integer -strict $answer]} {
            if {0 <= $answer && $answer < [llength $values]} {
                return $answer
            }
        }
    }
}

proc print_liberty {newvalues} {
    global lines close_braces
    puts [join $lines \n]

    puts "values ( \\"
    foreach elem $newvalues {
        puts [format "\"%s\", \\" [join $elem {, }]]
    }
    puts ");"

    for {set i 1} {$i <= $close_braces} {incr i} {
        puts [format %c 125]
    }
}

######################################################################
# define DSL
proc timing {label script} {
    lappend ::lines [format "timing %s %c" $label 123]
    incr ::close_braces
    uplevel 1 $script
}

proc rise_constraint {label script} {
    lappend ::lines [format "rise_constraint %s %c" $label 123]
    incr ::close_braces
    uplevel 1 $script
}

proc index_3 {args} {
    global idx3
    foreach item $args {
        lappend idx3 [string map {( "" \" "" , "" ) ""} $item]
    }
}

proc values {args} {
    global vals
    foreach set [lrange $args 1 end-1] {
        lappend vals [split [regsub -all {\s*,\s*} $set { }]]
    }
}

rename unknown system_unknown

proc unknown args {
    lappend ::lines "[join $args];"
}

######################################################################
if {$argc == 0} {
    puts "usage: $argv0 filename.lib"
    exit
}

set libfile [lindex $argv 0]
if {![file exists $libfile]} {
    error "no such file: $libfile"
}

main $libfile

Running it:
$ tclsh liberty.tcl test.lib  > newlib.lib
 0. 0.084
 1. 0.84
 2. 3.36
 3. 8.4
 4. 13.44
select an index_3 value: 0

$ cat newlib.lib
timing () {
related_pin : clk;
timing_type : setup_rising;
rise_constraint (constraint_template_5X5) {
index_1 ("0.01, 0.05, 0.12, 0.2, 0.4");
index_2 ("0.005, 0.025, 0.06, 0.1, 0.3");
values ( \
"1.1, 2.1, 3.1, 4.1, 5.1", \
"6.1, 7.1, 8.1, 9.1, 10.1", \
"11.1, 12.1, 13.1, 14.1, 15.1", \
"16.1, 17.1, 18.1, 19.1, 20.1", \
"21.1, 22.1, 23.1, 24.1, 25.1", \
);
}
}

You do lose all your nice indentation, but you get the values you want.
Comment: For your reading, Tcl syntax can be completely described on one page: http://tcl.tk/man/tcl8.5/TclCmd/Tcl.htm -- read that a few times to help you understand my code.
Comment: WOW!!! This is absolutely astounding stuff. You're a magician! I think it would have taken me a few eons to code this. I'm more of a hardware description coder (Verilog etc). Anyway, I will have to read up on the commands - some it them are very alien to me. Thanks so much your time and effort, I'm very appreciative. I will follow up with queries..

I tried running it in activetcl, somehow it crashes...hmm.
Answer: This is STF (Synopsys Technology File) or simple known as [dot]lib
liberty
I did work on these files on TCL platform but by use of some Parsers.
although many parser are available here is one for you.
I suggest to use parsers for handling [dot]lib as many times the file formate gets corrupt and is unable to read by primetime or other lib tool.
liberty_parser
there is another one available in C language too .. google vlsicad ucsd edu + liberty parser


Text processing pipeline for updating many source code files

Question: I need to take unspecified set of source code files and do the following operations:

If I find a declaration of certain type (basically, if I find a keyword in line based on a Regexp or simple search pattern) I need to add certain pre-defined fixed lines before that line.
I need to add standard comment above everything in the file
I need to perform cleanup of some lines or set of lines based on some criteria, for simplicity, based on Regexp search.

I have to run this on Windows machine, as source code will be .NET and Java, and as such I need tools that can be installed on Windows, but I am OK for instance with Cygwin etc.
Answer: there are three questions, not just one - hope that you are not going to set up a minus for a one suggestion only :)
If you know Linux shell, and I guess Cywin is the same shell basics, then the second question.
To add a standard comment on the top of each file, just
cat StdCommentFile SourceFile.h >> tempDir/SourceFile.h
In other words, you need to add a tempDir for the output. Of course if the standard heading is very simple, a small sed command could be OK.
The cleaning based on the regular expressions might need to be done with sed command too. I am a bit high level, since the details are not there :) Does that cleanup mean removal of entire lines or just a few phrases.


Definition algebra over commutative ring (injectivity needed?)

Question: I'm aware that many differently looking definitions exist for an algebra over a commutative ring $A$. For me, the most natural definition of an algebra over a commutative ring $A$ consists of a tuple $(B, f)$ where $B$ is a ring (not necessarily commutative) and $f: A \to B$ is a ring homomorphism satisfying $f(A) \subseteq Z(B)$, i.e. $bf(a) = f(a)b$ for all $a \in A, b \in B$.
Some authors seem to require that the homomorphism $f$ is also injective, such that $A$ can be embedded into $B$. I can't seem to find whether this assumption is common.
In my undergraduate thesis I want to use that, for an ideal $I$ of a commutative ring $A$ and some $a \in A$ fixed, there exists a unique $A$-algebra homomorphism
$$
\phi : A[X] \to A/I
$$
which maps $X$ to $a + I$. For this I need to be able to consider $A/I$ as an $A$-algebra, which I only can when I drop the injectivity of $f$. Is this clear to a mathematical audience, without having to define what I see as an algebra in my paper?
Comment: This injectivity assumption is clearly useless and very bad. You certainly want every ring to be a $\mathbb{Z}$-algebra.
Comment: Really, some authors require $f$ to be injective?  I have never seen that...
Comment: I'm not sure if this counts, but when Shafarevich talks about finite maps of varieties, the integral $k$-algebra extensions they correspond to are always injective, which definitely causes confusion; see, e.g., [this MO answer](http://mathoverflow.net/a/184925/33088).
Comment: When using the word 'extension' I find it a more natural assumption that the maps need to be injective.
Answer: Every ring can be considered as a $\mathbb{Z}$-algebra and, of course, the (unique) homomorphism is not necessarily injective.
It would be indeed quite strange that a quotient of an $A$-algebra is not necessarily an $A$-algebra, which is exactly the example you make.
Of course, if $A$ is a field, then the homomorphism is injective, provided the algebra is not the trivial ring.
Comment: The answer I had secretely been hoping for, thanks a lot!


Why UIImage View showing wrong Size of Image

Question: I have image files of dimension 1900x1200. In my code I try to load it as
<code>UIImage *image = [UIImage imageNamed:imageName];
</code>
When I try to run this code in iPhone Simulator (Retina Display) my images look out of proportion. I tried to print
image.size.width and image.size.height
and the value I get is 950x600.
What I am doing wrong. Please help me out.
Answer: The answer is actually simple: the UIImageView (and the underlying UIImage) are using scale factor 2.0. That is your 1900x1200 pixels image correspond to 950x600 points image with scale factor 2 on retina display. You can double check the UIImage's scale property.
Comment: How to download an image with exact scale size with SDWebImage?


Topological Circle as a Quotient (trivial quesion)

Question: Can someone explain me why does the unit circle can be expressed as the following quotient :
$ \mathbb{R} / \mathbb{Z} $ ?
Thanks !
Comment: @NilsMatthes: How on earth do you get two circles?
Comment: @Harald, yeah sorry, fixed that. Too early in the morning here in Germany for me ;)
Answer: Well, consider the map $x \mapsto \mathrm{e}^{2 \pi i x}$ from $[0,1)$ to $\mathbb{S}^1$. Prov that is is bijective. This may give you a hint.
Answer: Short answer: Think of the unit circle as residing in the complex plane, and consider the map $x\mapsto e^{2\pi ix}$. Now realize that $x_1$ and $x_2$ map to the same value if and only if $x_1-x_2\in\mathbb{Z}$. Or, in more group theoretic language: The map is a continuous group homomorphism of the additive group $\mathbb{R}$ onto the multiplicative group of the circle, and the kernel is $\mathbb{Z}$.


wp e-commerce product single page not working

Question: I am using wp e-commerce plugin & also used gold cart plugin for more functionality but my  product inner page not working.
There is use in grid view of products and files from plugin where being put in my own theme.
I put my wp e-commerce plugin > wpsc-components > theme-engine-v1 > templates files in my theme but it still not working. Can any one having idea about this? so please help me.
page url:-
http://www.eddingtonhousenursery.co.uk/wordpress/products-page/perennials/achillea-terracotta/
Comment: I you remove `products-page/` from the url, the page will load.
Answer: It seems you template file is not in the right place.
Please make sure you have moved all the plugin template files in the theme.
Best way to debug is print a string and die. Now check if your printed string appears on the page.


Input Password - HTML5, Intel xdk, Cordova

Question: Estou tendo um problema com o input type="password".
No Android o input password fica sempre iniciando com a letra maiúscula. Como faço para quando clicar no input a primeira letra já aparecer minúscula?
No IOS ele funciona corretamente. Segue o código que estou utilizando:
<input type="password" name="senha" value="">

Answer: Junior, tente usar o autocapitalize como off
<input type="password" name="senha" value="" autocapitalize="off">


Add a new colum with a button to field collection

Question: I'm trying to add a button or a link (< a href=''>< /a> or a < button>< /button>) in a new column in my field collection. For the moment i have 3 fields added by the administration of my field collection. I searched for add a button by the administration but we can't. what the best tips to add a button in a new column of my field collection ?
Do i need to "foreach" all element of my field collection and add a new element with the HOOK_form_alter ?
Here an example with the actual view (i cut the "delete line button" here so there aren't on the picture):

And now what i wanna create :

The "button" can be a link and after with css i style him.
PS : the button appear on edit content form
Thanks !
Comment: It looks like you are using Bootstrap type theme? By button, what do you mean? Do you want to add  tag?
Comment: i mean add a button like  or a  to go to another page
Answer: Use the Link Module to add a link field to your field collection

With this module links can be added easily to any content types and profiles and include advanced validating and different ways of storing internal or external links and URLs. It also supports additional link text title, site wide tokens for titles and title attributes, target attributes, css class attribution, static repeating values, input conversion, and many more.

And add the relevant bootstrap class to the output through the manage display UI.
Comment: Will OP be able to construct links using tokens?
Comment: Can't think of any reason why not @J.Reynolds, though admittedly I haven't tried it
Comment: Ok i will try the "link" module ! after i will need to add the bootstrap class with Hook form alter right ?
Comment: The link module actually lets you specify a class for the link through the UI (in the Manage Display screen), so you shouldn't have to write any code for this at all
Comment: Ok the module is working but i want to show the button in edit content form and not in the view of the content :/ there is a solution with "link" module ? thanks


Difference in number of frame rates on different browsers

Question: Suddenly, a simple animation has a different frameRate in chrome (approx 30fps) and firefox (approx 7fps).
Is that a right behavior ? And could i adjust framRate in my animation function ?
Comment: How you draw your code? Do you use `requestAnimationFrame()`? (Also, FPS always depends on the browser's speed and the quality of your code)
Comment: How are you checking the fps?
Answer: try this out
<code>// variables accessible from within function(frame)
var frameCount = 0;
var currentSecond = 0;
var frameRate = 0;

// within function(frame), called with current time on each new frame
function updateFrameRate(time) {
    var second = Math.floor(time / 1000); // ms to integer seconds
    if (second != currentSecond) {
       frameRate = frameCount;
       frameCount = 0;
       currentSecond = second;
    }
    frameCount ++;
}
</code>
A simple implementation, with "frames in a 1s interval". You can smoothen it out using, say, frames in a 5s interval
you can find more here Animation, FPS, and KineticJS
Comment: Please see this thread on copy pasting answers from another question: http://meta.stackexchange.com/questions/78658/is-it-okay-to-copy-paste-answers-from-other-questions
Comment: I mean, how can i set both 30 fps to chrome and firefox directly.


Logic app, convert run query json results to usuable / readable output (html/excelish?)to use in Sentinel comments for enrichment

Question: i have created a logic app to use in Sentinel for enrichment and automation. In this case its an automation app for identity based incidents, namely 'unfamiliar signins' The app is actually working perfectly, however. in the app at one point i run a kql query with the 'run query and list results' action. The query works, i can see the output in json. Now, with i use the raw output value, and use that in my Sentinel commment i do see the results, only its not in the best format as you can imagine.
The query looks for sign in events based on the user-agent seen in the incident, filters on use-agent and looks for events where the user agent has been seen in the past, does a summary based on application,useragent, and shows the results, this is perfect for incident inrichment. The results can vary, sometimes there are no results, because the user agent has not been seen in the past days/weeks, other times there are 4 rows with results, so its pretty dynamic.
I am looking for a way to make the output results of the query, which is in JSON, more usable, more readable, maybe in a html format and use that in the incident, or something else more readable.
the query i am using:
SigninLogs
| where UserPrincipalName == "<the upn variable>" // Replace with the specific user's UPN
| where UserAgent == "<the user-agent variable>" // Replace with the specific User-Agent string you're interested in
| where TimeGenerated between(ago(31d)..ago(1d)) // Filters to records from the last 30 days, excluding today
| summarize Count = count() by UserAgent, UserPrincipalName, AppDisplayName
| where Count > 0
| order by Count desc // Optional: Orders the results by count in descending order
| project Count, AppDisplayName, UserAgent, UserPrincipalName

I have found several posts online where this is being explained more or less, but or i am having trouble understanding or the instructions are incorrect ;)
Is there someone that can give me some good advise on this ?
regards!
your friend
Tried to play with array variables and such, but having trouble reflecting the internet instructions to my situation.
Comment: It would help if you could provide examples of data in the current format (json) and in the desired format.
Comment: To better assist you, could you please provide more details about what you've already tried to solve the issue? Any code snippets, error messages, or specific steps you've taken so far would be helpful. This information will help the community better understand your problem and provide more accurate solutions.
Answer:
only its not in the best format as you can imagine.I am looking for a way to make the output results of the query, which is in JSON, more usable, more readable, maybe in a html format and use that in the incident, or something else more readable.

Yes, In Logic apps there are many ways to make it into a readable format and below is my design:

Output:

Showing Readable data using CSV Table Connector:

Showing Readable data using HTML table Connector:

Not only this there are many connectors in Logic apps to design tables .


Find area of two overlapping circles using monte carlo method

Question: Actually i have two intersecting circles as specified in the figure
i want to find the area of each part separately using Monte carlo method in Matlab .
The code doesn't draw the rectangle or the circles correctly so
i guess what is wrong is my calculation for the x and y and i am not much aware about the geometry equations for solving it so i need help about the equations.

this is my code so far :
n=1000;
%supposing that a rectangle will contain both circles so :
% the mid point of the distance between 2 circles will be (0,6)
% then by adding the radius of the left and right circles the total distance
% will be 27 , 11 from the left and 16 from the right
% width of rectangle = 24

x=27.*rand(n-1)-11;
y=24.*rand(n-1)+2;
count=0;

for i=1:n

if((x(i))^2+(y(i))^2<=25 && (x(i))^2+(y(i)-12)^2<=100)
count=count+1;
    figure(2);
    plot(x(i),y(i),'b+')
    hold on

elseif(~(x(i))^2+(y(i))^2<=25 &&(x(i))^2+(y(i)-12)^2<=100)
    figure(2);
    plot(x(i),y(i),'y+')
    hold on

else
     figure(2);
    plot(x(i),y(i),'r+')

end

end

Comment: What exactly is your question? The vague *"I need help"* does not explain what you want. Please describe what the code is doing, both correct and incorrect. Please describe what you want. The [help] pages give lots off advice on how to write a good question. See also [mcve].
Comment: " i need help about that " was about the geometric equations ,, any ways i edited it , thanks for the notice
Answer: Here are the errors I found:
x = 27*rand(n,1)-5
y = 24*rand(n,1)-12

The rectangle extents were incorrect, and if you use rand(n-1) will give you a (n-1) by (n-1) matrix.
and
first If:
(x(i))^2+(y(i))^2<=25 && (x(i)-12)^2+(y(i))^2<=100

the center of the large circle is at x=12 not y=12
Second If:
~(x(i))^2+(y(i))^2<=25 &&(x(i)-12)^2+(y(i))^2<=100

This code can be improved by using logical indexing.
For example, using R, you could do (Matlab code is left as an excercise):
n = 10000
x = 27*runif(n)-5
y = 24*runif(n)-12
plot(x,y)

r = (x^2 + y^2)<=25 & ((x-12)^2 + y^2)<=100
g = (x^2 + y^2)<=25
b = ((x-12)^2 + y^2)<=100
points(x[g],y[g],col="green")
points(x[b],y[b],col="blue")
points(x[r],y[r],col="red")

which gives:
Comment: Thanks alot Sir , that was really helpful :)
Comment: Would anyone happen to have the formula for the area of the intersection area in this answer? I've plotted it but I haven't been able to find a solid answer anywhere on how to calculate the area numerically.
Answer: Here is my generic solution for any two circles (without any hardcoded value):
function [ P ] = circles_intersection_area( k1, k2, N )
%CIRCLES_INTERSECTION_AREA Summary...
%   Adnan A.
x1 = k1(1);
y1 = k1(2);
r1 = k1(3);

x2 = k2(1);
y2 = k2(2);
r2 = k2(3);

if sqrt((x1-x2)*(x1-x2) + (y1-y2)*(y1-y2)) >= (r1 + r2)
    % no intersection
    P = 0;
    return
end

% Wrapper rectangle config
a_min = x1 - r1 - 2*r2;
a_max = x1 + r1 + 2*r2;
b_min = y1 - r1 - 2*r2;
b_max = y1 + r1 + 2*r2;

% Monte Carlo algorithm
n = 0;
for i = 1:N
    rand_x = unifrnd(a_min, a_max);
    rand_y = unifrnd(b_min, b_max);

    if sqrt((rand_x - x1)^2 + (rand_y - y1)^2) < r1 && sqrt((rand_x - x2)^2 + (rand_y - y2)^2) < r2
        % is a point in the both of circles
        n = n + 1;
        plot(rand_x,rand_y, 'go-');
        hold on;
    else
        plot(rand_x,rand_y, 'ko-');
        hold on;
    end
end

P = (a_max - a_min) * (b_max - b_min) * n / N;

end

Call it like: circles_intersection_area([-0.4,0,1], [0.4,0,1], 10000) where the first param is the first circle (x,y,r) and the second param is the second circle.
Answer: Without using For loop.
    n = 100000;
    data = rand(2,n);
    data = data*2*30 - 30;
    x = data(1,:);
    y = data(2,:);
    plot(x,y,'ro');
    inside5 = find(x.^2 + y.^2 <=25);
    hold on
    plot (x(inside5),y(inside5),'bo');
    hold on
    inside12 = find(x.^2 + (y-12).^2<=144);
    plot (x(inside12),y(inside12),'g');
    hold on
    insidefinal1 = find(x.^2 + y.^2 <=25 & x.^2 + (y-12).^2>=144);
    insidefinal2 = find(x.^2 + y.^2 >=25 & x.^2 + (y-12).^2<=144);
    % plot(x(insidefinal1),y(insidefinal1),'bo');
    hold on
    % plot(x(insidefinal2),y(insidefinal2),'ro');
    insidefinal3 = find(x.^2 + y.^2 <=25 & x.^2 + (y-12).^2<=144);
    % plot(x(insidefinal3),y(insidefinal3),'ro');
    area1=(60^2)*(length(insidefinal1)/n);
    area3=(60^2)*(length(insidefinal2)/n);
    area2= (60^2)*(length(insidefinal3)/n);


Running 64-bit executable in System32 from 32-bit code

Question: I am trying to run an executable from a 32-bit C# app (on a 64-bit OS), but I'm getting "The system cannot find the file specified" probably because wsqmcons.exe does not exist in C:\Windows\SySWOW64. The file does exist in System32. What is the best way to run wsqmcons.exe from code, if possible?
<code>Process p = new Process();
p.StartInfo.Arguments = "-f";
p.StartInfo.FileName = @"C:\Windows\System32\wsqmcons.exe";
p.Start();
p.WaitForExit();
Verify.AreEqual(0, p.ExitCode);
</code>
Comment: For a quick-n-dirty fix, try changing the the path to: `@"C:\Windows\Sysnative\wsqmcons.exe"`.
See http://support.microsoft.com/kb/942589 for additional details/pointers that are more oriented to the Win32 API, but may help in .NET as well.
Comment: @Michael According to http://msdn.microsoft.com/en-us/library/aa384187%28VS.85%29.aspx it's recommended, so it's not so quick-n-dirty after all! I believe you should post this comment as an answer.
Answer: You will need to turn off the file system redirection on your 32 bit process with the Wow64DisableWow64FsRedirection and the re-enable it with Wow64RevertWow64FsRedirection.
Answer:
A 64-bit executable file located under %windir%\System32 cannot be
  launched from a 32-bit process, because the file system redirector
  redirects the path to %windir%\SysWOW64. Do not disable redirection to
accomplish this; use %windir%\Sysnative instead. For more information,
  see File System Redirector.

Quote from http://msdn.microsoft.com/en-us/library/windows/desktop/aa384203%28v=vs.85%29.aspx


Can you multiply if all you know is squaring?

Question: I am interested in this problem as a novelty; in the direction of constructing known things (which are usually taken to be somewhat fundamental) in terms of other, less-orthodox-to-call 'fundamental' things.
In particular, suppose the function $s(x) = x^2$ is taken to be known, or "effectively calculable"
(consider this a problem over the integers). I am interested then if it is possible to recover $f(x,y) = xy$ given only in terms of $s$ and other "simpler" operations, like addition or negation.
One attempt is given by the identity $(x+y)^2 -x^2 - y^2 = 2xy$. If the map $x \mapsto x/2$ were effectively calculable then we would be done in this case, but somehow this is unsatisfying to me. I would like to not 'give up' any amount of the "higher order" division or general multiplication operations.
I believe I can formalize my question in the following way.

Let $P \subseteq \mathbb Z [x,y]$ be the smallest collection of integer polynomials satisfying the following conditions.
\begin{equation*}
    \begin{array}{l}
      x \in P,\, y \in P\\
      p(x,y),\, q(x,y) \in P \implies mp(x,y) + nq(x,y) \in P \quad \forall m,n \in \mathbb Z\\
      p(x,y) \in P \implies (p(x,y))^2 \in P
    \end{array}
\end{equation*}
Question: is $xy \in P$?

We can see that it is the case that $2xy \in P$, but I wonder if removing the extraneous scalar $2$ will be impossible.
Comment: You should also consider $mx+ny+p$.
Comment: Are you asking if one can compute $xy$ just with a fixed number of $+,-,x^2,mx+ny$ ?
Comment: That sounds about right, yes
Comment: This can be summarized as $x^2,mx+ny$.
Comment: Isn't this just a special case of the [polarization identity](https://en.wikipedia.org/wiki/Polarization_identity) in one dimension?
Answer: No polynomial formed with linear combinations only does contain the term $xy$.
A polynomial obtained by squaring a linear combination has an even coefficient of $xy$.
The property "has an even coefficient of $xy$" is preserved both by linear combinations and squarings.
(This is also true of affine combinations.)
Comment: Thank you! Very simple, elegant counterargument
Answer: Instead of working in a situation where $x \mapsto x/2$ is not 'effectively calculable', why not work in a situation where it is impossible - i.e., in characteristic $2$?
By reducing modulo $2$, you may consider all your polynomials as elements of $\mathbb{F}_2(x,y)$. Squaring is the Frobenius endomorphism in characteristic $2$; i.e., we have
$$
(p + q)^2 = p^2 + q^2
$$
for any polynomials $p,q \in \mathbb{F}_2(x,y)$. From this, we can deduce by induction that any polynomial in your set $P$, when reduced modulo $2$, is a linear combination of powers of $x$ and $y$ (indeed, a linear combinataion of terms of the form $x^{2^a}$ or $y^{2^b}$). $xy$ is not of that form when reduced modulo $2$.


Concerning UTF-8 email addresses

Question: If I'm parsing an inbound email via php, and the address (through php) looks something like this:
=?UTF-8?B?5rWL6K+V5Zyw5Z2A?= <æµè¯å°å@server.tld>

And it's supposed to actually look something like this:
测试地址 <测试地址@server.tld>

Any idea how I could quickly convert from one to the other? Might be staring me right in the face, but regardless, not quite seeing it at the moment.
Regarding where this is occurring. Basically, assuming an arbitrary email is sent to a catch-all handled via a php script, the php script, on execution, reads the to header of this email:
$a = imap_open({inbox}FOO,[username],[password]);
$x = [mail index];
$headers = imap_header($a,$x);
$data = $headers->toaddress;

$data is what I'm concerned with.
Answer: try this:--
see this url-
PHP decode GB2312
https://www.php.net/manual/en/function.iconv-mime-decode-headers.php
A better way to convert would be to use iconv, see http://www.php.net/iconv -- example:
<?php
$myUnicodeString = "Åäö";
echo iconv("UTF-8", "ISO-8859-1", $myUnicodeString);
?>

Above would echo out the given variable in ISO-8859-1 encoding, you may replace it with whatever you prefer.
another solution
<?php
$headers_string = <<<EOF
Subject: =?UTF-8?B?UHLDvGZ1bmcgUHLDvGZ1bmc=?=
To: firstname.lastname@example.com
Date: Thu, 1 Jan 1970 00:00:00 +0000
Message-Id: <email@example.com>
Received: from localhost (localhost [127.0.0.1]) by localhost
    with SMTP id example for <firstname.lastname@example.com>;
    Thu, 1 Jan 1970 00:00:00 +0000 (UTC)
    (envelope-from firstname.lastname@example.com)
Received: (qmail 0 invoked by uid 65534); 1 Thu 2003 00:00:00 +0000

EOF;

$headers =  iconv_mime_decode_headers($headers_string, 0, "ISO-8859-1");
print_r($headers);
?>

Comment: i have improve my answer please take a look and added another solution. see the Subject: =?UTF-8?B?UHLDvGZ1bmcgUHLDvGZ1bmc=?=
Comment: Testing it out as we speak! So far, so good. Sorry, I'm a little slower than the rest of you, lol. I'll reply again to confirm results in a minute or two. ^.^
Comment: Testing this solution out, I'm actually not entirely sure if it's working or not. I think I might be doing something wrong on my end, perhaps.

To confirm: The example supplied (Subject) in the solution is returning "Prüfung Prüfung" for me - I assume this is correct?

I replaced your example with my $data, to see if it'd return expected results, but it returned an empty variable instead of an array(). I'm not sure if maybe this means my copy of php is manging the data, or something else..?

Directly base64_decoding()'ing my $data returns "测试地å€". Not sure if this is correct or not.
Comment: I have mention one more url in my answer. I am sure about it it is prefect answer according to your question.
Comment: ^ Above comment regarding secondary solution. First solution output seems to be identical to input - I don't notice any changes. :-/

Seems rather tricky. Solution looked good, and I'm hopeful for it, because I honestly don't know any solutions myself, but I think I might still be doing something wrong maybe.

Edit: Checking URL now. Posted this reply before I saw your follow-up comment.
Comment: see above the url I am sure about it it is prefect answer according to your question
Comment: Had additional unrelated complications that slowed things down a bit, but URL suggested seems to solve the issue. Thanks for the help, Abid. :)

Concerning UTF-8 email addresses


Is there a way to protect my PHP script from being edited?

Question: I've been working on a PHP class to build into a redistributable API. I'm trying to determine a way to effectively do a licensing check. At this point, I have the script check the URL that the code is running on, do a handshake on my server to see if there's a key for that URL, and then check the key against a local key in a config file.
My question is, is there a way to make this particular PHP file un-editable for future users? My predicament is that, while I do this check now, any future user of the code could easily go into my core class and swap my returns from the auth check method to be true no mater what the server response is.
Any ideas?
Thanks!
Comment: iconcube encoder or core zend should be what you need
Answer: You could do so by encoding your script with IonCube for example. There is a small fee per encoded file (I think it's calculated by LOC). But then you will have to distribute a license file with your software and the IonCube decoder (which is free) will be part of your software requirements list.
Comment: As usual, the community here is just chalk full of good answers. Thanks @Veseliq. I was aware of Zend, but had no interest in such an expensive encoder. IonCube seems to be a fairly good competitor.
Comment: Always glad to help. This very same community is helping me everyday as well :)


Why am I getting black spacers after text in html?

Question: So, I followed this tutorial: https://css-tricks.com/text-blocks-over-image/ for inserting text on an image. The code is shown below:

.image{
  position: relative;

}

h2{
  position:absolute;
  top: 200px;
  left: 200px;
  width: 100%;
}

h2 span {
   color: white;
   font: bold 24px/45px Helvetica, Sans-Serif;
   letter-spacing: -1px;
   background: rgb(0, 0, 0); /* fallback color */
   background: rgba(0, 0, 0, 0.7);
   padding: 10px;
}

h2 span.spacer {
   padding:0 5px;
}

A Movie in the Park:
Kung Fu Panda

But when I run the code I have these black things at the end of the first line and starting of the second line as a result of adding the spacing. I think its because of padding twice somehow.
These are not there in the tutorial on the website. Why do I have these and how can I remove them?
Thanks
Answer: Add following to your CSS:
h2 span.spacer {
    padding: 0 5px;
    background-color: rgba(0,0,0,0);
}

Seems like your background-colors are overlapping.
Comment: why are they overlapping?
Comment: You have a colored background with transparence and place a spacer with also transparent background-color. These two colors get combined to one color.
Comment: what if i have an image at the back of the text? it still doesn't work. do the background and spacing still get combined?
Answer: You should have transparent background for spacer btw:
h2 span.spacer {
   padding:0 5px;
   background: transparent;
}

Comment: why did they not do this in the link I provided? and why does their work properly but not mine?
Comment: Because the span inside h2 has color already, which is rgba(0, 0, 0, 0.7) or 70% opacity of black. So the span spacer double it.
Answer: Just make them invisible:
.spacer {
    opacity: 0;
}

Answer: You can also write 'transparent' like this:
h2 span.spacer {
    padding: 0 5px;
    background: transparent;
}


Update multiple workstations descriptions via powershell

Question: I am working in Powershell to update multiple workstation descriptions from a .csv file. The .csv file has contents looks something like this:
Workstation,Description
workstation1,description1
workstation2,description2

I want my script to look at the first column to get the workstation name then use Win32_OperatingSystem class and update the description of the computer with the information in the second column. Here is my code:
$myarray = Import-CSV -Path "workstations.csv" -Delimiter ','
$wmi = Get-WmiObject -ClassName Win32_OperatingSystem -cn $myarray.Workstation
$wmi.Description = $myarray.Description
$wmi.Put()

The problem I am running into is that when Powershell gets to the $wmi.Description = $myarray.Description line it is telling me "The property 'Description' cannot be found on this object."
Not sure where to go from here. I know the description is getting saved to the variable because I can output the contents of the variable with write-host.... Any tips or suggestions? It seems like a simple script.
Answer: The reason for the error is because the line below is retrieving an array of objects from the computers specified.
$wmi = Get-WmiObject -ClassName Win32_OperatingSystem -cn $myarray.Workstation

you can try using foreach in $myarray and the code should work fine
Full code:
$myarray = Import-CSV -Path "workstations.csv" -Delimiter ','
foreach ($x in $myarray)
{

$wmi = Get-WmiObject -ClassName Win32_OperatingSystem -cn $x.workstation
$wmi.Description = $x.description
$wmi.Put()
}

Comment: Could you elaborate a little more on what you mean by "in" `$myarray.workstation`? If I edit the above to contain the foreach I still get the same error.
Comment: @Ozar I have updated the answer can you check with full code?
Comment: You, sir, rock. Thank you so much.


Porque o vuejs diz que essa função não existe? se esta funcionando ok

Question: Tenho o seguinte codigo abaixo a função openDados(), ela funciona mais dentro do contexto de retorno da chamada socket não funciona porque..
lembro de ver algo semelhando tipo this = this algo do tipo mais não lembro aonde foi.... segue o codigo para os feras ajudar...
mounted () {
   this.openDados() // aqui funciona funciona
   this.socket.emit('find-user', {
      userid: this.id
    }, function (data) {
      this.openDados() // está func que não funciona
      console.log(data)
      if (data.length <= 0) {
        console.log('entrou aqui') // teste e chega aqui
      } else {
        const payload2 = {
          id: data.userid,
          name: data.name,
          display: data.display,
          descript: data.descript,
          avatar: data.avatar
        }
        this.USER(payload2)
      }
    })
  },
  methods: {
    ...mapMutations('config', ['SET_DISPLAY_LEFT']),
    ...mapMutations('user', ['USER']),
    mostrarImagem (imagem) {
      if (imagem !== null) {
        return process.env.API_S3 + '/wbynet/' + imagem
      } else {
        return 'statics/boy-avatar.png'
      }
    },
    openDados () {
      this.openDadosConf = true
    },

o erro que recebo
Answer: Isto acontece porque o this dentro de socket é outro contexto. O método que você chama pertence ao "this" externo. Desta forma, antes de chamar "this.socket...", crie algo como:
const self = this;

Então, dentro do socket você chama:
self.openData();

E pra tudo o mais que se referir ao "this" externo. Faça o teste e diga se resolveu. Sucesso pra voce;!
Comment: Obrigado, Salvo !!! Gratidão


CSS show line will filtering

Question: I have used an example of a timeline layout as you can see in the JSFiddle. I have added filter buttons and when I use them the line in the middle gets cut off. Can someone help me with this?
See:
https://jsfiddle.net/tc2axpkf/11/
Click on the button TEST or GROEP and see that the line in the middle gets cut off.
.timeline:before {
top: 0;
bottom: 0;
position: absolute;
content:" ";
width: 2px;
background: #ffffff;
left: 50%;
margin-left: -1.5px;
}

Answer: You are dealing with floatting elements that need to be cleared.
 .timeline {
    list-style: none;
    padding: 10px 0;
    position: relative;
    font-weight: 300;
    overflow:hidden; /* should be enough */
}

see for infos : https://css-tricks.com/all-about-floats/#article-header-id-4

If you are in a situation where you always know what the succeeding element is going to be, you can apply the clear: both; value to that element and go about your business. This is ideal as it requires no fancy hacks and no additional elements making it perfectly semantic. Of course things don't typically work out that way and we need to have more float-clearing tools in our toolbox.

In your case, you can use :

The Overflow Method
The Overflow Method relies on setting the overflow CSS property on a parent element. If this property is set to auto or hidden on the parent element, the parent will expand to contain the floats, effectively clearing it for succeeding elements. This method can be beautifully semantic as it may not require an additional elements. However if you find yourself adding a new div just to apply this, it is equally as non-semantic as the empty div method and less adaptable. Also bear in mind that the overflow property isn't specifically for clearing floats. Be careful not to hide content or trigger unwanted scrollbars.
Answer: As I can see the .timeline-inverted section is removed from the dom.
After this your .timeline collapses because of floated elements.
Try to add an clearfix:
.timeline:after {
  content: "";
  display: block;
  clear: both;
  height: 0;
}

CSS show line will filtering


Java socket programming: Is it possible for the server socket to initiate a communication?

Question: 1, my server has a ServerSocket to accept client connection
2, my client create a socket and and send message to the server.
3, the server gets the message and reply to the client.
So it's always the client that initiate the communication.
Say some time later after initial client connection, the server wants to send another message to the client, is it possible to do so?
Answer: If you keep the socket connection from the client open on the server yes. The client would need to be always listening for data coming in.
Otherwise the client will have to poll in intervals to the server if the connection on the server is closed after a message is sent.
Comment: True, but you would have to maintain a list of clients on the server end to connect to. All depends on what you want I suppose.
Comment: +1: Or you could let the "client" have a server port which allows the "server" to connect to the client.  However, the first suggestion of maintaining a connection whenever the client is available is much better (and much much faster and possibly more efficient)
Answer: The client must always initiate the connection, that's pretty much the only distinction between a server and a client (a program can be both be a server and a client).
However once a connection is established, as long as it's not closed either side can send data to the other (assuming the other side is listening for data).


Prove that $5\cos A+3\cos\left(A+\frac{\pi}{3}\right)+3$ lies between $-4$ and $10$

Question: After much of solving, I arrive at the expression
$$\frac{1}{2}\left[13 \cos A-3\sqrt{3}\sin A +6\right].$$
I can't conclude anything from this.
Please help. Hints are welcome.
Answer: Hint: By the Cauchy-Schwarz inequality,
$$ \left| 13\cos A-3\sqrt{3}\sin A\right| \leq \sqrt{13^2+27} = 14. $$
Answer: $\frac{1}{2}\left[13 \cos A-3\sqrt{3}\sin A +6\right]$
with $ \left| 13\cos A-3\sqrt{3}\sin A\right| \leq \sqrt{196} = 14 $
u get to:
$\frac{1}{2}\left[13 \cos A-3\sqrt{3}\sin A +6\right] \leq \frac{1}{2}\left[14 +6\right] = 10$


Como definir a função tangente no python?

Question: Gostarei que alguém me ajudasse a definir a função tangente usando o python com restrições em pontos onde a função não existe(pi/2 + k*pi(k inteiro). Desde já obrigado.
Comment: E qual é exatamente sua dúvida ou dificuldade? Já estudou sobre o módulo `math`?
Comment: Você também colocou as tags [tag:matplotlib] e [tag:graphics]. Você quer plotar o gráfico da tangente com os limites ou apenas usar os valores?
Answer: Use a biblioteca math. Ela tem um comando chamado de math.tan(x), na qual ela retorna o arco tangente de x em radianos.
Exemplo:
import math

math.tan(30)

>>> -6.40533119665


Entity Framework model doesn't work after an AppPool Recycle

Question: We're using Entity Framework version 5.0. The classes were created from Model-First, but now are code-first. Examples:
public class tTableName: EntityTypeConfiguration<EntityName> {
    public tTableName() {
        // Primary Key
        this.HasKey(t => t.ID);

        // Properties
        this.Property(t => t.Column2)
                .IsRequired()
                .HasMaxLength(255);

        this.Property(t => t.Column5)
                .HasMaxLength(5);

        // Table & Column Mappings
        this.ToTable("tTableName");
        this.Property(t => t.ID).HasColumnName("ID");
        this.Property(t => t.Column2).HasColumnName("Column2");
        this.Property(t => t.Column3).HasColumnName("Column3");
        this.Property(t => t.Column4).HasColumnName("Column4");
        this.Property(t => t.Column5).HasColumnName("Column5");
        this.Property(t => t.Column6).HasColumnName("Column6");
    }
}

public class EntityName{
    public int ID { get; set; }
    public string Column2{ get; set; }
    public string Column3 { get; set; }
    public DateTime? Column4 { get; set; }
    public string Column5 { get; set; }
    public int? Column6 { get; set; }
}

In my DataContext class, I've overridden OnModelCreating and pass in the tTableName classes one at a time. This was all created for me by reverse-engineering my Database (Schema-First? Database-First?).
So the Database exists, the schema is populated, and we aren't using any EF migrations or initializers.
Database.SetInitializer<DataContext>(null);
The Problem
When our application starts up, everything loads and initializes just fine. I do a database query in my Application_OnStart just to make sure it does work. But, at some point after running fine for 24 hours, after an Application Pool Recycle, the application stops working. Restarting IIS, Recycling the App Pool, Even rebooting the machine has no effect. The error that I get is:

The entity TypeName is not part of the model for the current context.

This isn't so nice. It worked a moment ago. But nothing I do can get it to work again. EXCEPT uploading a fresh binary or making a change to the /bin folder. This causes something to reset so that Entity Framework is convinced to rebuild the model. When tracing the code, and looking at my logs, it appears that Entity Framework has somehow "forgotten" the database model and is just skipping the ModelCreating step when the application is restarting.
I assume that Entity Framework is building up the model and then emitting it or saving it to something which is why it doesn't feel the need to call ModelCreating. According to this answer, Entity Framework (as of 4.0) will cache the model, but only reuse it when creating a new DbContext within the same AppDomain. So what could cause the model to become corrupted? How can I force model-rebuilding in Application_OnStart?
Answer: As it turns out, I did have some other code, but mainly it was because I was separating my Entities and my EntityModelConfiguration classes into separate projects (we had just too many) and sometimes the ASP.NET runtime would load my assemblies in a different order, and sometimes it wouldn't have loaded them yet (or at all). So I used this answer to pre-load all of my assemblies from my /bin folder during application startup.


Help understanding Vanishing and Exploding Gradients

Question: I am following deeplearning.ai's videos on Coursera. I have a couple of questions regarding vanishing and exploding gradients. The following is Prof Andrew Ngs lecture slides:

From what Prof Ng says in the lectures, having large weights leads to large gradients and having having small weights leads to small gradients and both of these are troublesome for training. My questions are:
1.) Assuming my activation function is Sigmoid (say g(x)), having very large weights (x) will lead to small gradients and slow learning through Gradient Descent.
If, x is close to 0, then my gradients will be very large. Can this lead to overstepping in Gradient Descent (akin to a large Learning Rate)? I am guessing this is not the case as the gradient is due to the nature of the cost function while the learning rate is a parameter we set.
2.) If my activation function is ReLU, the slope will be constant for any x>0 and 0 for x<0. So, where is the notion of exploding or vanishing gradients coming from for ReLU? I thought ReLUs solve the vanishing and exploding gradient problem.
Thanks in advance.
Answer:
1.) Assuming my activation function is Sigmoid (say g(x)), having very large weights (x) will lead to small gradients and slow learning
  through Gradient Descent. If, x is close to 0, then my gradients will
  be very large. Can this lead to overstepping in Gradient Descent (akin
  to a large Learning Rate)? I am guessing this is not the case as the
  gradient is due to the nature of the cost function while the learning
  rate is a parameter we set.

"If, x is close to 0, then my gradients will be very large" --> "very" is very relative here. It will be 0.25 if x=0.

2) If my activation function is ReLU, the slope will be constant for
  any x>0 and 0 for x<0. So, where is the notion of exploding or
  vanishing gradients coming from for ReLU? I thought ReLUs solve the
  vanishing and exploding gradient problem.

You can still lose the signal with ReLUs because the gradient will be 0 for negative inputs.
EDIT:

Alright. I understand how the vanishing gradients occur. But how is it ever possible to get "exploding" gradients if my gradients are never >1?

Consider the following neural network sketch:

For example, the partial derivative of the loss of the weight $w_{1,1}^{1}$, doesn't just rely on the derivative of the ReLU function. I.e., the partial derivative would be
$$\begin{aligned} \frac{\partial l}{\partial w_{1,1}^{(1)}}=\frac{\partial l}{\partial o} \cdot \frac{\partial o}{\partial a_{1}^{(2)}} & \cdot \frac{\partial a_{1}^{(2)}}{\partial a_{1}^{(1)}} \cdot \frac{\partial a_{1}^{(1)}}{\partial w_{1,1}^{(1)}} \\ &+\frac{\partial l}{\partial o} \cdot \frac{\partial o}{\partial a_{2}^{(2)}} \cdot \frac{\partial a_{2}^{(2)}}{\partial a_{1}^{(1)}} \cdot \frac{\partial a_{1}^{(1)}}{\partial w_{1,1}^{(1)}} \end{aligned}$$
Comment: the vanishing gradient problem occurs if you have a long chain of multiplications that includes values smaller than 1. Vice versa, if you have values greater than 1 in the chain of multiplications, you will have exploding gradients. ReLU helps somewhat (except dead ReLUs) with the vanishing gradient, but as you can see from the example I included, the chain of multiplications is not just about the relu derivative, so yes, you can still have exploding gradients if several of the values there are greater than 1 and you have a very deep network
Comment: With exploding gradients, you will basically overshoot. I.e., say there is a local loss minimum you want to converge too, instead of taking a small step into that direction, you take a huge leap that is so large that you end up in a worse position than you started from.
Comment: Alright. I understand how the vanishing gradients occur. But how is it ever possible to get "exploding" gradients if my gradients are never >1?
Comment: Sure, I edited the answer above to explain
Comment: I see. So vanishing gradients are an issue as they slow down learning, but what is the issue with exploding gradients? Also, does using ReLU activations solve the vanishing and exploding gradients problems?
Comment: Thank you. You've cleared most of my doubts. Final question: vanishing gradients slow down learning. What is the issue with exploding gradients?
Comment: @resnet I have a question about learning rate, as Batch norm original paper (https://arxiv.org/pdf/1502.03167.pdf), they said  " too-high learning rate may
result in the gradients that explode or vanish". As I know the vanishing problem mean the Gradient is small, let assume the optimizer is SGD, so the bigger learning rate should be better, right? Could you explain about this or I should open new question?
Comment: You can potentially counteract the vanishing problem by choosing a higher learning rate. However, keep in mind that you typically use the same learning rate everywhere in the network. So while you avoid vanishing gradient problems in one layer by choosing a very large learning rate, you now introduce problems elsewhere in the network by having a learning rate that is too large (and it could then backfire and also cause exploding gradient problems)
Comment: @resnet "the partial derivative of the loss of the weight w11,1, doesn't just rely on the derivative of the ReLU function. I.e., the partial derivative would be". what does it mean? Can you please elaborate?


Understanding Varnish: Can it cache 10,000 static pages on a site?

Question: First off, I use: APC, W3TC, PHP5, Wordpress 3.8, Apache 2.2 and am getting a lot of "Cache Misses"
Still don't really understand Varnish that well. Here are some stats I got from the Unixy backend from a few minutes running:
client_conn: 3744
client_drop: 0
client_req: 3910
cache_hit: 914
cache_miss: 2347
I changed my Varnish memory to 3GB of RAM (out of 8 total).
Is it normal that Varnish only caches pages that are hit frequently? So let's say you have a site with 10,000 pages, is Varnish capable of keeping all those in the cache or does it only cache a certain amount of frequently requested pages?
So my question really is how I can improve Varnish to cache more pages and keep them in the cache for a long time (mostly static pages that dont need to be refreshed a lot)
Comment: Varnish only caches pages by default that have been hit more than $x time in $y minutes, doesn't cache pages with cookies and obviously can only cache pages that have already been requested.
Answer: 3GB should be enough, seriously. If you actually hit the limit, you won't have cache miss but cache drop which is, in your case: 0.
So the memory is probably not the problem.
The varnish memory usage is pretty straight forward, it uses not much memory to work, and most of the memory is used to store objects, object is about your webpages, so an html page of 50kb cached in varnish will use about ~50kb in varnish.
HOWEVER, if your object and therefore its associated object hash is different, for example because there are different query parameters (not impacting the contact but used to track for example), therefore each object will be cached separately.
here is the default vcl to compute hash for object:
sub vcl_hash {
  hash_data(req.url);
  if (req.http.host) {
    hash_data(req.http.host);
  } else {
    hash_data(server.ip);
  }

  return (hash);
}

As you can see it is based on the full url and the host. It exists a querystring-vmod to sort and/or removes query params to increase the hit count.
This may be a good start to check if your object are not considered different by varnish (in most case it is because of query params).
But in your case I actually suspect that you've cookies, and/or wrong cache control headers.
By default, varnish will ignore all requests with cookies, increasing your cache miss count.
sub vcl_recv {
  // ...

  if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */

    return (pass);
  }
}

If you actually know what you're doing you may remove cookies from requests and force cache of your object but be careful, you may cache an admin page this way which will be delivered to guests.
For a example, you may define which url should be cached and set a Cache-Control: public header, and in your vcl, detect a public Cache-Control header, remove cookies, and cache.
But again, be careful.

EDIT: You may be interested in this article: Varnish and Wordpress on the Varnish docs
Comment: Thank you for the thorough answer, my cache hit rate is about 50% now cache_hit: 13505
cache_miss: 21895 --- I removed some javascript that added cookies just to make sure varnish is working properly. I will now inspect cache control headers
Comment: The headers I set via htaccess are "Cache-Control: max-age=3600, public" and Pragma: public
Answer: On top of the Unixy Varnish config I added this after researching various configurations, maybe some of it is redundant but my cache HIT ratio is a lot better now like 60% and I managed to get the CPU load from 3 down to 0.10 - 0.50
wordpress backend
if (!(req.url ~ "wp-(login|admin)")) {
                unset req.http.cookie;
        }

other files
  if ( req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$" ) {

        unset req.http.cookie;
  }

cookies
if (req.http.Cookie) {
    set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "\1");

    if (req.http.Cookie == "") {
        remove req.http.Cookie;
    }
}

purging
if (req.request == "PURGE") {
return (lookup);
}

Also 3GB is a little overkill, I may change it back 1 or 2GB .. even with 10,000 pages or more, but I will let it run for 24 hour and see


Reynould equation and partial answer

Question: I've been working on solving an ODE equation in trying to find the pressure gradient along a hydrostatic bearing, as far as literature goes,
the given ode
$\frac{\partial }{\partial r}\left(\frac{h^3}{12 \eta
    }\text{}r\text{ }\frac{\partial p}{\partial r}\right)=0$
with a simply given solution of;
$p(r)\text{:=}\frac{\text{pt} (\log (r)-\log (R))}{\log (\delta )-\log (R)}$
With $p(\delta )=\text{pt},\text{ }p(R)=0$ as the Initial values.
For me, it seems to look like a simple double integrated ode
$\frac{\partial ^2p}{\partial r^2}=\text{ }0$
, with a solution of:
$p(r)\to \frac{\text{pt} R-\text{pt} r}{R-\delta }$
and infact, when evaluating with mathematica, I get the same answer as by hand.
My solution is a very simply linear one...and since I'm dealing with typically non-linear systems I can't imagine it's as simple as I have solved it.
So, to ask the question, how did they get their answer?
Thanks for the help!
Comment: I'm not sure if I understand your question. The given ODE is indeed linear, assuming $h$ and $\eta$ are constants. Your simplified ODE is a common approach people take to understand difficult equations (typically more applied in PDE's) by considering the highest order derivative term.
Comment: ahh, yes you're right the ode is linear...Ahhh i meant the solution i have...is quite literally a straight line when plotted...the pressure gradient in a hydrostatic bearing should drop as one approaches the edges non linearly...
Answer: $$\frac{\partial }{\partial r}\left(\frac{h^3}{12 \eta
    }\text{}r\text{ }\frac{\partial p}{\partial r}\right)=0$$
Integrating once with respect to $r$ we get
$$\frac{h^3}{12\eta}\frac{\partial{p}}{\partial{r}}=\frac{c_{1}}{r}$$
for some constant $c_{1}$. Integrating the second time,
$$\frac{h^3}{12\eta}p(r)=c_{1}\ln{(r)}+c_{2}$$
or, using $A=\frac{12c_{1}\eta}{h^{3}}$ and $B=\frac{12c_{2}\eta}{h^{3}}$
$$p(r)=A\ln{r}+B$$
using the boundary conditions we have
$$B=-A\ln{R}$$
$$A=\frac{pt}{\ln{\delta}-\ln{R}}$$
Comment: Ahh so that's how...thanks! But why not factor in d/dr first then simplify h^3/12eta before integrating ?...or is that against order of operations ?
Comment: Yeh, $\frac{h^3}{12\eta}$ commutes with $\frac{d}{dr}$, thus, as long as it is not zero you can divide it out.

Reynould equation and partial answer


Limit number of concurrent user logins in Windows Server 2008 Active Directory

Question: Is there the possibility to limit Active Directory users' max concurrent login sessions?
I've read many articles and discussions about the solution, but none of them seem to be working.
Many had suggested UserLogin script that doesn't work in Windows Server 2008. Some other suggested CConnect that is not good enough. It's also very complicated. Some others have introduced UserLock that should be paid for.
It's wondering that Windows Server 2003 DOES have the feature (wile as a third-party), but Windows Server 2008 doesn't have!
One of the articles I've read: http://www.edugeek.net/forums/windows-server-2008-r2/61216-multiple-logins.html
Comment: I don't think that Microsoft really see this as a problem that needs solving - lets face it, outside of education and other specialist situations what's the business case for enabling this? As someone who works in education, however, I do have the need and to be honest we just trialed userlock (mentioned in the edugeek thread I believe), discovered it works really well, placed an order and then got on with our day... Not the answer you were looking for, I appreciate, but the best I have. You can stop multiple logins with scripting of course, for free, but that just causes more problems...
Answer: There is no built-in functionality in any version of Windows Server to do what you're asking for. You're going to have to write something yourself or procure third-party software.
The only thing even remotely close built-in would be to limit the computers the account can log in to (through ADUC for instance).  In essence, if you took the time to limit an account to a single workstation, then it almost gives you the functionality.  But it's a PITA to administer and is more of a workaround to the request.


UITableView with Hierarchical Categories -Swift iOS

Question: I have 3 categories:

Sneakers
Pants
Shirts

Each category can have a number of different name brands eg.
1.Sneakers > Nikes, Adidas, Reeboks
2.Pants > Lees, Levi's
3.Shirts > Ralph Lauren, Abercrombie
My users pick from a category and then they can add as many name brands as they want to it. After they've chosen the category and put whatever brand names they want, I need 2 tableViews (or collectionViews). The first tableView should show the category they picked and the 2nd tableview should show the brands they named in that category.
My problem is sorting by category. I know how to display all of the names they choose, but not the categories leading to the names.
The issue I'm having is if a user choose the Shirts category twice, the tableView to display it displays the shirt category twice, when it should only display it once.
I found this from Apple but it doesn't show the examples in Swift.
Any similar StackOverflow q/a's in Swift are fine as an answer.
Comment: So after they update the category, you are saying the Shirt is showing twice, which means your array containing category, contains Shirt twice, right? if so, use NSOrderedSet to remove the duplicate. Hope it helps
Comment: @Janmenjaya I've never used it before but I'll try it and let you know. Thanks!
Comment: I have updated the answer, check it to see the use of NSOrederedSet, to remove duplicate.
Comment: Can you show your code?
Answer: If your array contains duplicate in category.
I hope it must be containing duplicate, as you can see the duplicate in table view, so use the NSOrderedSet to remove duplicate.
Here is the demo code demonstrating use of NSOrderedSet
Objective C Version:
NSArray *category = [NSArray arrayWithObjects:@"Sneakers", @"Pants", @"Shirts",@"Pants", @"Shirts", nil];

NSLog(@"Before using ordered set : %@", category.description);

// iOS 5.0 and later
NSArray * newCategoryArray = [[NSOrderedSet orderedSetWithArray:category] array];

NSLog(@"After using ordered set  : %@", newCategoryArray.description);

Swift Version :
let category = ["Sneakers", "Pants", "Shirts","Pants", "Shirts"];
NSLog("Before using ordered set : %@", category.description);

let newCategoryArray = NSOrderedSet(array: category).array;

NSLog("After using ordered set  : %@", newCategoryArray.description);

Output :
Before using ordered set : (
    Sneakers,
    Pants,
    Shirts,
    Pants,
    Shirts
)

After using ordered set  : (
    Sneakers,
    Pants,
    Shirts
)

Happy coding ...
Comment: i couldn't find the equivalent method in swift
Comment: @LanceSamaria, I have added the Swift equivalent code. Check the answer and let me know.
Comment: @thanks I'll try it in a few. I'm about to post a more detailed explantation of what I'm looking for. I'm not sure if your method will work with it but I will try it. Here's the link http://stackoverflow.com/questions/39663089/uitableview-category-array-drill-down-swift-ios


Raw or orthogonal polynomial regression?

Question: I want to regress a variable $y$ onto $x,x^2,\ldots,x^5$. Should I do this using raw or orthogonal polynomials? I looked at the question on the site that deals with these, but I don't really understand what's the difference between using them.
Why can't I just do a "normal" regression to get the coefficients $\beta_i$ of $y=\sum_{i=0}^5 \beta_i x^i$ (along with p-values and all the other nice stuff) and instead have to worry whether using raw or orthogonal polynomials? This choice seems to me to be outside the scope of what I want to do.
In the stat book I'm currently reading (ISLR by Tibshirani et al) these things weren't mentioned. Actually, they were downplayed in a way.
The reason is, AFAIK, that in the lm() function in R, using y ~ poly(x, 2) amounts to using orthogonal polynomials and using y ~ x + I(x^2) amounts to using raw ones. But on pp. 116 the authors say that we use the first option because the latter is "cumbersome" which leaves no indication that these commands actually do two completely different things (and have different outputs as a consequence).
(third question) Why would the authors of ISLR confuse their readers like that?
Comment: @gung That actually confused me more than it helped. Previously I thought that I should just go with orthogonal polynomials, because that seemed to be the right way, but in that answer raw polynomials are used. Suprisingly, everyone on the net is screaming "RTFM", but there is actually not clear answer, when to use what. (Your link also doesn't give an answer to this, just an example, when orth. pol. go wrong)
Comment: Unless you are working in some physical or engineering domain that states the response will be a quintic polynomial, almost surely the right approach is not to do polynomial regression in the first place.  Invest your degrees of freedom in a spline or something that would be *far* more flexible and stable than the polynomial fit.
Comment: @Sycorax I know that `poly` has something to do with orthogonal polynomials and I(x^2) doesn't (though I don't know the details) - but still, why would the authors of ISLR then recommend a method that does not work ? It seems very misleading if both command seem to do the same, but only one actually is ok.
Comment: @gung I looked at the documentation of `poly` and spent already a while with this problem, but I can't figure out why poly(x,2) and x+I(x^2) make a difference? Could you please enlighten me here in the comments, if the question is offtopic?
Comment: @gung I did a completey re-edit of my question. This choice raw/orthogonal is confusing me even more  - previously I thought this was just a minor `R` technicality, that I didn't understand, but now it seems to be a fullblown stat problem that hinders me of doing coding a regression that should not be that difficult to code.
Comment: Why the downvote? I just completely re-edited my question.
Comment: Did you bother to read the help for `poly` which specifies that these are orthogonal polyomials? https://en.wikipedia.org/wiki/Orthogonal_polynomials
Comment: This question now seems on topic to me. I'm retracting my close vote.
Comment: Related: [How to interpret coefficients from a polynomial model fit?](http://stats.stackexchange.com/q/95939/)
Answer: I feel like several of these answers miss the point. Haitao's answer addresses the computational problems with fitting raw polynomials, but it's clear that OP is asking about the statistical differences between the two approaches. That is, if we had a perfect computer that could represent all values exactly, why would we prefer one approach over the other?
user5957401 argues that orthogonal polynomials reduce the collinearity among the polynomial functions, which makes their estimation more stable. I agree with Jake Westfall's critique; the coefficients in orthogonal polynomials represent completely different quantities from the coefficients on raw polynomials. The model-implied dose-response function, $R^2$, MSE, predicted values, and the standard errors of the predicted values will all be identical regardless of whether you use orthogonal or raw polynomials.
The coefficient on $X$ in a raw polynomial regression of order 2 has the interpretation of "the instantaneous change in $Y$ when $X=0$." If you performed a marginal effects procedure on the orthogonal polynomial where $X=0$, you would get exactly the same slope and standard error, even though the coefficient and standard error on the first-order term in the orthogonal polynomial regression are completely different from their values in the raw polynomial regression. That is, when trying to get the same quantities from both regressions (i.e., quantities that can be interpreted the same way), the estimates and standard errors will be identical. Using orthogonal polynomials doesn't mean you magically have more certainty of the slope of $X$ at any given point. The stability of the models is identical.
In the example below, I fit a raw polynomial model and an orthogonal polynomial model on the same data using polynomials of order 3. To get a parameter with the same interpretation as the slope on the second-order (squared) term in the raw model, I used a marginal effects procedure on the orthogonal model, requesting the slope when the predictor is equal to 0.
data("iris")

#Raw:
fit.raw <- lm(Petal.Length ~ Petal.Width + I(Petal.Width^2) +
                  I(Petal.Width^3), data = iris)
summary(fit.raw)

#> Coefficients:
#>                  Estimate Std. Error t value Pr(>|t|)
#> (Intercept)        1.1034     0.1304   8.464 2.50e-14 ***
#> Petal.Width        1.1527     0.5836   1.975  0.05013 .
#> I(Petal.Width^2)   1.7100     0.5487   3.116  0.00221 **
#> I(Petal.Width^3)  -0.5788     0.1408  -4.110 6.57e-05 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.3898 on 146 degrees of freedom
#> Multiple R-squared:  0.9522, Adjusted R-squared:  0.9512
#> F-statistic: 969.9 on 3 and 146 DF,  p-value: < 2.2e-16

#Orthogonal
fit.orth <- lm(Petal.Length ~ stats::poly(Petal.Width, 3), data = iris)

#Marginal effect of X at X=0 from orthogonal model
library(margins)
summary(margins(fit.orth, variables = "Petal.Width",
                at = data.frame(Petal.Width = 0)))
#> Warning in check_values(data, at): A 'at' value for 'Petal.Width' is
#> outside observed data range (0.1,2.5)!
#>       factor Petal.Width    AME     SE      z      p  lower  upper
#>  Petal.Width      0.0000 1.1527 0.5836 1.9752 0.0482 0.0089 2.2965

Created on 2019-10-25 by the reprex package (v0.3.0)
The marginal effect of Petal.Width at 0 from the orthogonal fit and its standard error are exactly equal to those from the raw polynomial fit (i.e., 1.1527). Using orthogonal polynomials doesn't improve the precision of estimates of the same quantity between the two models.
The key is the following: using orthogonal polynomials allows you to isolate the contribution of each term to explaining variance in the outcome, e.g., as measured by the squared semipartial correlation. If you fit an orthogonal polynomial of order 3, the squared semipartial correlation for each term represents the variance in the outcome explained by that term in the model. So, if you wanted to answer "How much of the variance in $Y$ is explained by the linear component of $X$?" you could fit an orthogonal polynomial regression, and the squared semipartial correlation on the linear term would represent this quantity. This is not so with raw polynomials. If you fit a raw polynomial model of the same order, the squared partial correlation on the linear term does not represent the proportion of variance in $Y$ explained by the linear component of $X$. See below.
library(jtools)
data("iris")

fit.raw3 <- lm(Petal.Length ~ Petal.Width + I(Petal.Width^2) +
                  I(Petal.Width^3), data = iris)
fit.raw1 <- lm(Petal.Length ~ Petal.Width, data = iris)

round(summ(fit.raw3, part.corr = T)$coef, 3)
#>                    Est.  S.E. t val.     p partial.r part.r
#> (Intercept)       1.103 0.130  8.464 0.000        NA     NA
#> Petal.Width       1.153 0.584  1.975 0.050     0.161  0.036
#> I(Petal.Width^2)  1.710 0.549  3.116 0.002     0.250  0.056
#> I(Petal.Width^3) -0.579 0.141 -4.110 0.000    -0.322 -0.074

round(summ(fit.raw1, part.corr = T)$coef, 3)
#>              Est.  S.E. t val. p partial.r part.r
#> (Intercept) 1.084 0.073 14.850 0        NA     NA
#> Petal.Width 2.230 0.051 43.387 0     0.963  0.963

fit.orth3 <- lm(Petal.Length ~ stats::poly(Petal.Width, 3),
               data = iris)
fit.orth1 <- lm(Petal.Length ~ stats::poly(Petal.Width, 3)[,1],
               data = iris)

round(summ(fit.orth3, part.corr = T)$coef, 3)
#>                                Est.  S.E.  t val. p partial.r part.r
#> (Intercept)                   3.758 0.032 118.071 0        NA     NA
#> stats::poly(Petal.Width, 3)1 20.748 0.390  53.225 0     0.975  0.963
#> stats::poly(Petal.Width, 3)2 -3.015 0.390  -7.735 0    -0.539 -0.140
#> stats::poly(Petal.Width, 3)3 -1.602 0.390  -4.110 0    -0.322 -0.074

round(summ(fit.orth1, part.corr = T)$coef, 3)
#>                                    Est.  S.E. t val. p partial.r part.r
#> (Intercept)                       3.758 0.039 96.247 0        NA     NA
#> stats::poly(Petal.Width, 3)[, 1] 20.748 0.478 43.387 0     0.963  0.963

Created on 2019-10-25 by the reprex package (v0.3.0)
The squared semipartial correlations for the raw polynomials when the polynomial of order 3 is fit are $0.001$, $0.003$, and $0.005$. When only the linear term is fit, the squared semipartial correlation is $0.927$. The squared semipartial correlations for the orthogonal polynomials when the polynomial of order 3 is fit are $0.927$, $0.020$, and $0.005$. When only the linear term is fit, the squared semipartial correlation is still $0.927$. From the orthogonal polynomial model but not the raw polynomial model, we know that most of the variance explained in the outcome is due to the linear term, with very little coming from the square term and even less from the cubic term. The raw polynomial values don't tell that story.
Now, if you want this interpretational benefit over the interpretational benefit of actually being able to understand the coefficients of the model, then you should use orthogonal polynomials. If you would prefer to look at the coefficients and know exactly what they mean (though I doubt one typically does), then you should use the raw polynomials. If you don't care (i.e., you only want to control for confounding or generate predicted values), then it truly doesn't matter; both forms carry the same information with respect to those goals. I would also argue that orthogonal polynomials should be preferred in regularization (e.g., lasso), because removing higher-order terms doesn't affect the coefficients of the lower order terms, which is not true with raw polynomials, and regularization techniques often care about the size of each coefficient.
Comment: @skan The coefficients on the orthogonal polynomials are uninterpretable. I would argue that understanding what $X^2$ means doesn't mean you know what the coefficient on $X^2$ means, but maybe you do (I wouldn't attempt to interpret it). If your goal is prediction rather than interpretation of the coefficients in your model, does it really matter? If you want to understand the relationship between X and Y, plotting it is better than trying to read coefficients, and the plot will be the same whether you use raw or orthogonal polynomials.
Comment: Excellent contribution.

I can't replicate your marginal results (the margin function pops an error about poly when I try to run your first block of code -- I'm not familiar with the margin package) -- but they are exactly what I expect. As a small suggestion -- you should include the output of the margin analysis on the raw model as well. Your argument is undermined (slightly) by the change in p-values from the summary to the margin functions (changing our conclusions no less!) -- which seems to be caused by using a normal instead of a t distribution. Your regularization point is excellent.
Comment: Thank you for the kind words. I think you have to include the `stats::` in the call to `poly()` in `lm()` for `margins` to recognize it (which is stupid). I wanted to focus my argument on the point estimates and standard errors, and I know there is a lot of extraneous and distracting information presented, but I hope the text illustrates my points.
Comment: @skan it doesn't matter. Both will give you identical predicted values of Y for each value of X with the same standard error. For computational reasons it might be better to use orthogonal, especially if you are dealing with very large or very small values.
Comment: @Noah.  I understand what are X and X^2 in a regular model, and how to interpret the coefficients.
But what are X and X^2 in a model that has been orthogonalized by poly()?
Comment: That's not it. I copied your code exactly, and you use `stats::poly()`. The error says `'degree' must be less than number of unique points` -- which doesn't help me much. Nevertheless, `margin()` is backing up provable statements so it is not important.
Comment: What if I want to fit this polynomial and use it to calculate the value of X that gives me the maximum Y. Would I need to use raw=T or raw=F to compute the polynomial?
Answer: I believe the answer is less about numeric stability (though that plays a role) and more about reducing correlation.
In essence -- the issue boils down to the fact that when we regress against a bunch of high order polynomials, the covariates we are regressing against become highly correlated. Example code below:
x = rnorm(1000)
raw.poly = poly(x,6,raw=T)
orthogonal.poly = poly(x,6)
cor(raw.poly)
cor(orthogonal.poly)

This is tremendously important. As the covariates become more correlated, our ability to determine which are important (and what the size of their effects are) erodes rapidly. This is typically referred to as the problem of multicollinearity. At the limit, if we had two variables that were fully correlated, when we regress them against something, its impossible to distinguish between the two -- you can think of this as an extreme version of the problem, but this problem affects our estimates for lesser degrees of correlation as well. Thus in a real sense -- even if numerical instability wasn't a problem -- the correlation from higher order polynomials does tremendous damage to our inference routines. This will manifest as larger standard errors (and thus smaller t-stats) that you would otherwise see (see example regression below). For this reason, we might choose to orthogonalize our polynomials before regressing them.
y = x*2 + 5*x**3 - 3*x**2 + rnorm(1000)
raw.mod = lm(y~poly(x,6,raw=T))
orthogonal.mod = lm(y~poly(x,6))
summary(raw.mod)
summary(orthogonal.mod)

If you run this code, interpretation is a touch hard because the coefficients all change and so things are hard to compare. Looking at the T-stats though, we can see that the ability to determine the coefficients was MUCH larger with orthogonal polynomials. For the 3 relevant coefficients, I got t-stats of (560,21,449) for the orthogonal model, and only (28,-38,121) for the raw polynomial model. This is a huge difference for a simple model with only a few relatively low order polynomial terms that mattered.
That is not to say that this comes without costs. There are two primary costs to bear in mind. 1) we lose some interpretability with orthogonal polynomials. We might understand what the coefficient on x**3 means, but interpreting the coefficient on x**3-3x (the third hermite poly -- not necessarily what you will use) can be much harder.
Second -- when we say that these are polynomials are orthogonal -- we mean that they are orthogonal with respect to some measure of distance. Picking a measure of distance that is relevant to your situation can be difficult. However, having said that, I believe that the poly function is designed to choose such that it is orthogonal with respect to covariance -- which is useful for linear regressions.
Comment: -1. The larger standard errors that you see on the lower order coefficients is a red herring. The lower order coefficients in your two models are estimating completely different things, so comparing their standard errors makes no sense. The highest order coefficient is the only one estimating the same thing in both models, and you'll see that the t statistic is identical whether the polynomials are orthogonal or not. Your two models are statistically equivalent in terms of fitted values, R^2, etc., they differ mainly just in interpretation of the coefficients
Comment: Re: the first point, sorry, I meant to refer to the t-stat of the highest-order term, not its coefficient. That predictor is scaled+shifted between models, so yes the coef changes, but it tests the same substantive effect, as shown by _t_
Comment: Re: the second point, the reason "the t-stats are substantially different" for the lower-order terms is, again, because they are estimating completely different things in the two models. Consider the linear effect: in `raw.mod` it estimates the slope of the curve at x=0, in `orthogonal.mod` it estimates the marginal slope (i.e., identical to `lm(y ~ poly(x,1))` where higher-order terms are omitted). There is no reason that estimates of these completely different estimands should have comparable standard errors. One can easily construct a counter-example where `raw.mod` has much higher t-stats
Comment: @JakeWestfall. I still think you're missing the multicollinearity. However, we seem to be talking past each other, and there is perhaps a solution. You say you can easily construct a counter-example, please do. I think seeing the dgp you have in mind would clarify a lot for me. At the moment the only things I've been able to come up with which might behave as you describe involve severe model misspecification.
Comment: @JakeWestfall, I don't think I agree with you.

First of all, running the code produces values which are different for all of the polynomial orders, not all but one -- in essence it takes the polynomial and does PCA on it. Secondly, and more importantly, the t-stats are substantially different -- running the code in my answer will confirm that -- functionally we are solving the multicollinearity problem.

You are right that fitted values, R^2, F-tests etc don't change. That in fact is a reason to orthogonalize -- it changes nothing *except our ability to detect polynomial terms*.
Answer:
Why can't I just do a "normal" regression to get the coefficients?

Because it is not numerically stable. Remember computer is using fixed number of bits to represent a float number. Check IEEE754 for details, you may surprised that even simple number $0.4$, computer need to store it as $0.4000000059604644775390625$. You can try other numbers here
Using raw polynomial will cause problem because we will have huge number. Here is a small proof: we are comparing matrix condition number with raw and orthogonal polynomial.
> kappa(model.matrix(mpg~poly(wt,10),mtcars))
[1] 5.575962
> kappa(model.matrix(mpg~poly(wt,10, raw = T),mtcars))
[1] 2.119183e+13

You can also check my answer here for an example.
Why are there large coefficents for higher-order polynomial
Comment: You seem to be using single precision floats and quoting them to quadruple precision! How did that happen? Except for GPUs, almost all statistical computation uses at least double precision.  E.g., in `R` the output of `print(0.4, digits=20)` is `0.40000000000000002`.
Answer: I would have just commented to mention this, but I do not have enough rep, so I'll try to expand into an answer. You might be interested to see that in Lab Section 7.8.1 in "Introduction to Statistical Learning" (James et. al., 2017, corrected 8th printing), they do discuss some differences between using orthogonal polynomials or not, which is using the raw=TRUE or raw=FALSE in the poly() function. For example, the coefficient estimates will change, but the fitted values do not:
# using the Wage dataset in the ISLR library
fit1 <- lm(wage ~ poly(age, 4, raw=FALSE), data=Wage)
fit2 <- lm(wage ~ poly(age, 4, raw=TRUE), data=Wage)
print(coef(fit1)) # coefficient estimates differ
print(coef(fit2))
all.equal(predict(fit1), predict(fit2)) #returns TRUE

The book also discusses how when orthogonal polynomials are used, the p-values obtained using the anova() nested F-test (to explore what degree polynomial might be warranted) are the same as those obtained when using the standard t-test, output by summary(fit). This illustrates that the F-statistic is equal to the square of the t-statistic in certain situations.
Comment: Comments should never be used as answers regardless of your reputation numbers.
Comment: Regarding your last point, this is true of non-orthogonal polynomials, too. The coefficient t-test is equal to the F-test comparing a model with the coefficient in it and a model without for all coefficients in regression (taken one at a time).


How do it get id of artist using spotifyr?

Question: I am trying to get ids of artist based on artist name, album, etc. cannot seem to find a way to do so. Please help me with it. I do not need the entire program spoon fed, just the part on how to return id either through album name or artist name or both.
All of your help is much appreciated.
I have looked into documentation in https://cran.r-project.org/web/packages/spotifyr/spotifyr.pdf still nothing
Answer: Example:
artist = "enya"
df = spotifyr::get_artist_audio_features(artist)
names(df)
 [1] "artist_name"                  "artist_id"                    "album_id"
 [4] "album_type"                   "album_images"                 "album_release_date"
 [7] "album_release_year"           "album_release_date_precision" "danceability"
[10] "energy"                       "key"                          "loudness"
[13] "mode"                         "speechiness"                  "acousticness"
[16] "instrumentalness"             "liveness"                     "valence"
[19] "tempo"                        "track_id"                     "analysis_url"
[22] "time_signature"               "artists"                      "available_markets"
[25] "disc_number"                  "duration_ms"                  "explicit"
[28] "track_href"                   "is_local"                     "track_name"
[31] "track_preview_url"            "track_number"                 "type"
[34] "track_uri"                    "external_urls.spotify"        "album_name"
[37] "key_name"                     "mode_name"                    "key_mode"
View(df)

You can see that field artist_id is returned.
In this example:
print(df$artist_id[1])
"6uothxMWeLWIhsGeF7cyo4"

I hope it helps.


Are GAM models linear in the parameters?

Question: Consider a GAM model, expressed in mgcv just to fix ideas:
my_model <- gam(y ~ ti(x1)+ti(x2) + ti(x1, x2), method= "REML")

The model is linear in the parameters, right? Each smooth is a linear combination of basis functions, which are independent of the data set (unless I use bs = "ad"). Thus the model is linear in the parameters, which are the coefficients of the basis functions. Right? And this should be true, whether or not there are interaction terms - it doesn't really matter. The only exception are adaptive smooths, because in that case the coefficients of the basis functions are themselves functions of the covariates (x1 and x2, in my example). Correct?
Answer: Yes, GAMs are linear in the parameters. If we ignore the estimation of smoothness parameters, once we have created the bases for all the covariates we want to fit a smooth effects of, a GAM is just plain old GLM with coefficients for individual basis functions.
This is also true in smooth interactions. The ti(x1, x2) term is just a tensor product basis formed by two univariate marginal bases and the resulting coefficients map to individual functions in the 2-d basis. (The ti() basis has had the main effects of the separate covariates removed when invoked with two or more covariates.)
Comment: @DeltaIV Ha! :-) Simon doesn't need any help from me; I'm just a very grateful user myself.
Comment: You're a guarantee :-) you should consider collaborating with Simon Wood for the third edition of his book...or write one yourself!


IUnity Container RegisterTypeforNavigation is not working in Prism 7

Question: I am having this problem when I installed prism 7

SCS1929    'IUnityContainer' does not contain a definition for 'RegisterTypeForNavigation' and the best extension method overload 'UnityExtensions.RegisterTypeForNavigation(IUnityContainer, string)' requires a receiver of type 'IUnityContainer'

Has anyone encountered this problem?
Answer: This would seem to be the result of you only partially updating your code. Due to a number of changes particularly with Unity, Prism 7 decoupled its API from any specific container. As a result, the extensions for RegisterTypeForNavigation no longer exist for the IUnityContainer, but instead for the new IContainerRegistry. When registering your Services and Views in Prism 7, you would do this in RegisterTypes(IContainerRegistry containerRegistry).
This actually provides a number of benefits including the fact that the documentation remains consistent for Prism users regardless of which container you're using. While you have not directly asked this question yet, I should mention that you should pay particular attention to how you are referencing Unity. The new maintainer for Unity changed a lot about the container which was a major break for users going from older versions of Prism to Prism 7 where we adopted the updated version of Unity. This included namespace breaks, as well as a complete change in the dependent package. If you are directly referencing the Unity Container please be sure to uninstall it and only reference the package that Prism does.


How to search all string in MySql that start with a number?

Question: I want to run a mysql query to give me all available results that start with a number.
SELECT * FROM users WHERE username LIKE number + '%'

I tried using
LIKE REGEXP '^[0-9]

but it is not working for me
Answer: You can use REGEXP:
SELECT *
FROM users
WHERE username REGEXP '^[0-9]'

SQL Fiddle Demo
Comment: Just barely faster than me!
Comment: This is the correct answer. It return the string that start with a number
Comment: @Mike -- Glad we could help!  Best of luck.
Answer: Your RegExp is wrong, and the way you are using the REGEXP operator is wrong.  Your code should be
WHERE username REGEXP '^[0-9].*'

SQLFiddle Example
Edit: I'm wrong about the RegEx portion.  I didn't realize you don't need the .*
Comment: Thank you for your help but this return any string that contains a number. @sgeddes got it right :)  Thanks