Live statistics from a log file

Question: I've got nginx running as a load balancer in front of several web servers. I'd like to be able to see live statistics with response time (maybe a histogram of the last 10 seconds?)
I think Varnish has something similar
Tail isn't very useful as there are about 1k requests per second
Do you know of any tool that can do this? (just so that I don't reinvent the wheel)
edit
I think I wasn't clear enough in the original post. We're already using collectd to grather various kinds of stats from all the servers, and we've got a nice interface to see those graphs live.
I'm looking for something similar to htop
Answer: I don't have a specific solution here, so hopefully someone else will chime in, but this looks like a job for RRDTool. You'd need to look for (or write) something which will pull the data you need from the log into the round-robin database, but the tool is set up to handle just what you're talking about, just the most-recent chunk of time-series data.
Answer: You know about the Nginx stats stub, right?  There are also some links at the bottom of that page related to Nginx monitoring techniques.  I'm not sure about getting the response time out of that, though.
Answer: Unfortunately, I don't know of any existing way to get live response time data. Given that I'd expect it to be a rare occasion when you need to check such data, it seems likely that the performance cost it requires (i.e. additional processing/logging per request) does not seem justified. I'd suggest simply logging the response time data to your logs (e.g. like this) and run a shell script to parse a portion of your logs (and output a summary) when you are interested in the data.
The live statistics are not too difficult to come by - but getting response time with them is harder. The approaches below only provide raw data - they do not graph it for you.
If you can live without response time (e.g. just req/s - and a breakdown of # reading, writing, active, total connections, etc), use the HTTP Stub Status Module. It provides a text output of information that can be easily be queried and parsed. (To use this module, Nginx needs to be compiled with --with-http_stub_status_module)
Sample Output:
Active connections: 291
server accepts handled requests
  16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

(The 3rd line is accepted connections, handled connections, accepted requests)
There is an interesting article on using this data to generate graphs with rrdtool, here.
If the response time data is particularly essential to you, it should be possible to modify the Stub Status Module. The code (only about 5kB actually) is in src/http/modules/ngx_http_stub_status_module.c. For instance, there is a modification here that has the module output aggregated counts of HTTP status codes returned.
Lastly, if you just want response time. You could save the data to Memcached using the HTTP Memc Module. For the purposes of your histogram, you would essentially create 2 keys per second - base the key on the time (e.g. $time_local) - one key to store a counter of the number of requests in that second, and one to store the average response time (or cumulative response time) - working with the $request_time and/or $upstream_response_time variables.
I am sure you can come up with an easy way of graphing the data, once you have it, but here are some possibly starting points:

Have a script pull the data (from stub, or memcached, etc) and generate ascii bars (e.g. like this), use watch to monitor the output of the script.
If your server has PHP (and you willing to use it (not the ideal scripting language)), consider modifying this script.
Finally, perhaps consider gnuplot (output as_ascii), although it seems excessive for the purpose.

Live statistics from a log file


Prove that the following collection is a $\sigma$-algebra of sets

Question: Let it be ($\mathrm{X}$,$\rho$) a metric space and let it be $\mathcal{S}$$\subseteq$$\mathcal{P}$($\mathrm{X}$) such that $A$$\in$$\mathcal{S}$ if and only if there is an open $\mathrm{U}$$\subseteq$$\mathrm{X}$ and for every $n$$\in$$\mathbb{N}$, there is a $E_n$$\subseteq$$\mathrm{X}$ such that $int(cl(E_n))=$$\varnothing$ in a such way that $A$$\bigtriangleup$$\mathrm{U}=$$\bigcup\{E_n:n\in\mathbb{N}\}$. Prove that $\mathcal{S}$ is a $\sigma$-algebra of subsets of $\mathrm{X}$.
I've already prove that $\mathcal{S}\neq\varnothing$ and that $\mathrm{X}\in\mathcal{S}$ but i don't have any idea of how to proof that $\mathcal{S}$ is closed under countable unions and that $\mathcal{S}$ is closed under set difference.
I know that i should use topological properties and set theorical operations but i'm not very good at that.
Answer: This is one of the classic $\sigma$-algebras (like the Borels sets). A set $E$ such that $\operatorname{int}(\operatorname{cl}(E)) = \emptyset$ is a "nowhere dense set". A countable union of nowhere dense sets is " a set of the first category" or a meagre/meager set. Intuitively, such sets are "small": in a complete metric space no open set can be of the first category. The $\sigma$-algebra you describe are those that are "almost open" (they differ by an open set by a meagre set. These sets have the property of Baire. Look in Oxtoby's book (Measure and Category, chapter 4) for a proof of being a $\sigma$-algebra, e.g. It's in fact the $\sigma$-algebra generated by the collection of sets that are open or meagre.


Google App Maker driver picker files

Question: In App Maker we have drive picker widget that will save files in current user google drive.
When a user uploads a file to his drive, the other users in the application should access the file from the uploaded URL. Or is there a option to configure a common drive for all users to save there files.
Answer: This problem should be eventually addressed by Team Drives, but it is not supported by App Maker at this time. Meanwhile, you can use Drive API (Server Script) to put uploaded files into a shared folder:
<code>// ID of the folder shared with all interested parties
var folderId = '9A18nXnY1C0amK0WCPI1WqMPObDj';
var folder = DriveApp.getFolderById(folderId);

var fileId = 'JI09KJkjIU7kjUYNdeIcNK1nFG8a';
var file = DriveApp.getFileById(fileId);

folder.addFile(file);
</code>
or explicitly grant permissions to the file:
<code>var fileId = 'JI09KJkjIU7kjUYNdeIcNK1nFG8a';
var file = DriveApp.getFileById(fileId);

file.addCommenter('email@example.com');
file.addViewer('firstname.lastname@example.com');
file.addEditors(['firstname.lastname@example.com', 'firstname.lastname@example.com']);
</code>
Comment: Thanks pavel, Appmaker does not support Team drives right
Answer: That's not really an answer but I can't comment due to the my lack of points. You can use team drive or shared folder depending on your needs. When the upload is complete, you can move the file. You can either bind it to OnDocumentSelect of widget or do it when the user clicks the final approval depending on your flow.
Alternatively, you can let people drop into that folder and use appmaker to read the contents of the folder and let user pick. I am not sure which one is better.
Comment: Oh Is it possible to use team drive in appmaker, how to configure it.
Comment: You can get a list of team drives but I don't currently see an option to add it as a parent to the file.

Google App Maker driver picker files


Use the new Facebook Photo Viewer (gallery) from iframe app

Question: I have an (iframe) app hosted on a tag on a Facebook Page. The app pulls (based on various conditions) photos from the albums of the page and displays them to the user. Is it possible to create "ajaxify" links on the photos so that when the user clicks on the photo, the photo will open in the new Facebook Photo Viewer (lightbox overlay) without navigating to the actually photo's URL?
Answer: Facebook's lightbox isn't part of the facebook api. You will have to implement your own lightbox, or download one from somewhere, to get this effect.
Comment: I was thinking of a more hackish way, such as triggering some facebook function from within the iframe. There might be same-origin issues, though.
Comment: This isn't possible if you don't own both domains. This may be of use to you for more info: http://stackoverflow.com/questions/702074/cross-domain-access-in-iframe-from-child-to-parent


How to interpret decreasing AIC but higher standard errors in model selection?

Question: I've got a problem choosing the right model. I have a model with various variables (covariables and dummy variables). I was trying to find the best size for this model, so I first started by comparing different models with AIC. From this it followed, that the minimum AIC was reached when allowing all variables to stay in the model (with the whole bunch to interact with all dummies). When I compute the summary of the model, all effects are absolutely not significant and the standard errors are very high. I was a bit confused, when comparing the "best" (on AIC) model with a smaller model with any interaction. The smaller model had small standard errors and nice p-values... But the AIC is higher compared to the big model. What might be the problem? Overspecification?
I really need help in this, because I have absolutely no idea how to handle this!
Thanks a lot
Comment: Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation?
Answer: The AIC and standard error measure different things, and if you are trying to minimize standard error, a cross-validation approach may be better to use.  Another alternative is the Bayesian information criterion (BIC), which is more parsimonious than the AIC.
Also, here's a good article comparing the relations between various evaluation metric for supervised machine learning: Data mining in metric space: an empirical analysis of supervised learning performance criteria.
Comment: I'd also suggest that your statement that BIC is more parsimonious than AIC needs to be expanded to clarify that BIC will typically choose simpler models than AIC, but that this by no means implies that the models that BIC chooses are "better". BIC fundamentally assumes that the "true" model is amongst the models under consideration, which is probably a stretch for most circumstances. AIC on the other hand simply attempts to minimize future error prediction (hence the connection to cross validation) and makes no claims regarding the "truth" of the selected model.
Comment: As I understand it, AIC and cross validation (at least, leave-one-out cross validation) are asymptotically equivalent, therefore your suggesting that the OP employ cross validation boils down to ignoring the uncross-validated SE and go with the AIC instead.
Comment: Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation?


Probability of a journey

Question: A trip from south east London to Southampton consists of three journeys: bus journey to Crystal Palace station, train journey from Crystal Palace to Clapham Junction, train journey from Clapham Junction to Southampton. Each journey can be delayed (d) or on time (o) with the following probabilities:
For the bus journey to Crystal Palace station: P(d)=2/3 and P(o)=1/3
For the train journey from Crystal Palace to Clapham Junction: P(d)=1/4 and P(o)=3/4
For the train journey from Clapham Junction to Southampton: P(d)=1/4 and P(o)=3/4 The lateness or not of any transport is independent of the lateness or not of any other transport.
Consider the following two events:
A={delays in the trip are due only to delays in the bus journey}
B={delays in the trip are due to delays in one or both of the train journeys}
Calculate the probabilities P(A), P(B).
Attempting this question:
I created a sample space where
S = {OOO, OOD, ODO, ODD, DDD, DOO, DOD, DDD}
P(A) = DOO
2/3*3/4*3/4 =3/8
P(B) = OOD, ODO, ODD,
1/3 * 3/4 * 1/4 + 1/3 * 1/4 * 3/4 + 1/3 * 1/4 * 1/4 = 7/48
Is this correct?
Also, if the probabilities were not independent how would I calculate the probability?
Answer: Yes, this is correct.  If the probabilities were not independent, they would have to tell you how the chance a train is late changes with whether the bus is late.  You would have to use the proper chance for the train being late in each of your multiplications.

Probability of a journey


Python unittests: how to get an AsyncMock to block until a certain point

Question: I have some async Python code I want to test, which include something like
async def task(self):
    while True:
        await self.thing.change()
        ...
        self.run_callbacks()

In my test file, I'm starting the task in the setUp() function like so:
def setUp(self):
   ...
   self.loop = asyncio.get_event_loop()
   self.task = self.loop.create_task(self.object_under_test.task())

then wanting to verify that the callbacks ran:
def test_callback(self):
    callback = mock.MagicMock()
    self.object_under_test.add_callback('cb1', callback)
    # ???
    callback.assert_called_once()

Where I'm stuck is how to mock thing.change() in the task. It needs to block until I tell it to (at ???) at which point it will return. If I mock it after starting the task, then it won't make any difference as the task is already waiting for the un-mocked function to return. If I mock it prior to starting the task, then it will always return something and not block.
Any suggestions how I might achieve this?
Answer: I've worked out how to synchronise using the asyncio.Event object and as side effect on the mock:
# my_task.py
import asyncio

async def async_task():
    while True:
        result = await trigger()
        await asyncio.sleep(0)

async def trigger():
    # Some asynchronous function you want to mock
    await asyncio.sleep(1)
    print('triggered')
    return "Trigger Completed"

# test_my_task.py
import asyncio
import unittest
from unittest.mock import patch, AsyncMock
import my_task

class TestAsyncTask(unittest.TestCase):

    def setUp(self):
        self.trigger_event = asyncio.Event()
        async def mocked_trigger():
            await self.trigger_event.wait()
            return "Mocked Trigger Completed"
        my_task.trigger = AsyncMock(side_effect=mocked_trigger)

    def test_async_task_with_mocked_trigger(self):
        loop = asyncio.get_event_loop()

        task = loop.create_task(my_task.async_task())

        loop.run_until_complete(asyncio.sleep(1))
        self.trigger_event.set()
        loop.run_until_complete(asyncio.sleep(0))

        self.assertGreaterEqual(my_task.trigger.await_count, 1)

if __name__ == "__main__":
    unittest.main()


kickstart ks.cfg: Where should `url --url` point?

Question: I have a kickstart file (ks.cfg) on a floppy (Old style). I am trying to install CentOS 5.4.
The top of my ks.cfg says this:
install

# Install from local cdrom or over the network.
#cdrom
url --url http://kickstart.example.org/pub/centos/5.4/

On the Apache server side, this command is failing with these 404s:
kickstart.example.org 192.168.16.180 - - [01/Jun/2010:17:24:30 -0700] "GET /pub/centos/5.4///disc1/.discinfo HTTP/1.1" 404 314 "-" "urlgrabber/3.1.0"
kickstart.example.org 192.168.16.180 - - [01/Jun/2010:17:24:43 -0700] "GET /pub/centos/5.4/repodata/repomd.xml HTTP/1.1" 404 316 "-" "urlgrabber/3.1.0 yum/3.2.22"

It seems that the value of my url doesn't match the directory structure on the server, but this is an exact mirror of the CentOS mirrors.
What should the value of url URL be? Should this only include the OS (/pub/centos/5.4/), or should it include the architecture (/pub/centos/5.4/os/x86_64 )? Can this be a variable (/pub/centos/5.4/os/$ARCH/)?
I see that Kickstart is trying to grab a file called 'repomd.xml', but why is it looking in '/pub/centos/5.4/repodata/repomd.xml', when these files actually exist at '/pub/centos/5.4/os/x86_64/repodata/repomd.xml' and other locations at '/pub/centos/5.4/*/$ARCH/repodata/repomd.xml'?
I don't see this documented or explained well in the RedHat 5 Installation Guide
Answer: I have a 5.3 and 5.4 serving on a custom ks.cfg file. My line looks like this:
url  --url http://long-path-to-your-distro/x86_64/5.3  -- or 5.4 -- no trailing slash, but it shouldn't matter.
and in there I have:
./
../
CentOS/
config/
EULA
GPL
images/
isolinux/
NOTES/
repodata/
RPM-GPG-KEY-beta
RPM-GPG-KEY-CentOS-5
TRANS.TBL

Comment: thanks for upvoting the answer. BTW, I just put out my notes on doing pxe boot for 5.4 Centos. They might help: http://notagrouch.com/linux/centos-54-pxe-boot.html
Answer: You should point it at the path that contains the .discinfo file and/or the repodata/ directory.
Comment: This works, but I wish the '--url' flag was explained better somewhere.
Comment: So I need to set something like 'url --url http://mirrors.kernel.org/centos/5.4/os/x86_64/' instead of 'url --url http://mirrors.kernel.org/centos/5.4/'?
Comment: If that's where `.discinfo` and `repodata/` are located, then yes.


How do I get rid of the white space between my adsense ad and home and next post buttons?

Question: I want to tighten up or get rid of the white space between my google ad and my home and older posts buttons any advice?
Answer: Use this snippet -
<code>.footer-inner {
    padding-top: 0;
}
.region-inner.main-inner {
    padding-bottom: 0;
}

div#blog-pager {
    margin-bottom: 0;
    padding-bottom: 0;
}
</code>
Comment: Add this too and see what happens.
Comment: im confused as what to type in


Como vejo as coordenadas do mouse na tela?

Question: Estou aprendendo sobre a lib pyautogui do Python e nisso gostaria de saber exatamente onde está o ponteiro do mouse para poder efetuar o click com precisão.
Answer: Basta usar a função .position() do pyautogui para obter o x e y do ponteiro em relação a tela. por exemplo:
<code>print(pyautogui.position())
</code>
Recomendo que utilize essa função com o sleep para dar tempo de você colocar o ponteiro em cima do botão.
Caso queira um click ainda mais preciso recomendo você utilizar a função .locateCenterOnScreen ou .locateAllOnScreen. Você consegue ver mais sobre essas funções na documentação do pyautogui:
https://pyautogui.readthedocs.io/en/latest/screenshot.html


How to retrieve data from set with drush_set_option() from within install profile

Question: I've been messing around with trying to pass varabiles to an install profile based on a signup form, as descripbed in this post. I'm able to get the data out of the form and into drush using hook_pre_provision_install(), but I'm not sure how to grab that data from within the install profile that was set with drush_set_option().
Answer: It appears the answer is simple: You can use drush_get_option().

Get the value for an option.
If the first argument is an array, then it checks whether one of the options exists and return the value of the first one found. Useful for allowing both -h, and --host-name.


Deleting the head of a linked list does not main the hash structure?

Question: Yet another data structure doubt. I will get straight to it.
This is what I have
use strict;
use warnings;
use Data::Dumper;

my $head= undef;
my $tail=\$head;

open FILE, "<datastored.txt" or die $!;

while (<FILE>){
    my $node = {
                "data" => $_ ,
                "next" => undef
            };
    $$tail=$node;
    $tail = \${$node->{"next"}};
};

# #1# Print full list #
print Dumper $head;

# #2# Delete the first node and display data of next node #
$head = $head->{next};
my $value = $head->{data};

and this is the output I get
$VAR1 = {
          'next' => \{
                        'next' => \{
                                      'next' => \{
                                                    'next' => \undef,
                                                    'data' => 'line 4'
                                                  },
                                      'data' => 'line 3
'
                                    },
                        'data' => 'line 2
'
                      },
          'data' => 'line 1
'
        };
Not a HASH reference at linkedlist.pl line 32, <FILE> line 4. **<<<< My Problem and hence my question?**

Note - the content of the file datastored.txt is simply
line 1
line 2
line 3
line 4

If I look at the print result of $head initially, it clearly is a hash within a hash and so on, so why is that after deleting the first node is the integrity of the data structure hampered? (See Error on last line of output)
Answer: it clearly is a hash within a hash and so on
Actually, it's clearly not. Each next holds a reference to a reference to a hash, not a reference to a hash. Notice all the "\"? You should be seeing
$VAR1 = {
  'next' => {
    'next' => {
      'next' => {
        'next' => undef,
        'data' => 'line 4'
      },
      'data' => 'line 3'
    },
    'data' => 'line 2'
  },
  'data' => 'line 1'
};

Comment: So how do i get the value out?
Comment: `$head = $head->{next};` is correct. The error is creating all those references to references to hashes in the first place.
Comment: isn that how i should create Linked Lists see http://stackoverflow.com/questions/10940533/meaning-of-next-in-linked-list-creation-in-perl/10940796#10940796
Comment: Nope. As your data dump shows, you got a slew of useless references to references to hashes. You shouldn't be seeing those "`\ `" in the dump.
Comment: Yeah I understand now, linked lists are killing me! Thanks for your help.
Answer: Perl references are tricky.  The dumper output notation \{ is showing that the {next} fields contain references to references to hashes.
The offending line is the update to tail.  Try
$tail = \$node->{"next"};

Comment: With the code change I posted, the extra level of references goes away, and your code to access the second item works fine.
Comment: O yes I see that now, so my follow up question would be, How do i get the value of the second node, and for any node of my choice?
Comment: Yeah I understand now, linked lists are killing me! Thanks for your help.
Comment: Two things, (1) its not home work, I am trying to learn Linked lists(check my profile) and see the questions, I am progressively getting stuck and need someone to help, I would consider my self lucky if I had some one giving me homework (cause the teacher would teach me eventually). I am referring "O'Reilly -Mastering Algos with Perl". (2) I appreciate you correcting the solution with your previous comment, but the solution provided by Gene worked and dint understand why not to make scalar references?


Let $F$ Be the Set of All Functions $f : \mathbb{R} \to \mathbb{R}$. Prove That $\preceq$ Is a Partial Order.

Question:
Let $F$ be the set of all functions $f : \mathbb{R} \to \mathbb{R}$. A relation $\preceq$ is defined on F by
$f \preceq g$ if and only if $f(x) \le g(x)$ for all $x \in \mathbb{R}$.
Prove that $\preceq$ is a partial order.

I know that, in order to prove that a relation (in this case, $\preceq$) is a partial order, we must prove that the relation satisfies three things:

Reflexitivity: If $a \in R$, then $(a, a) \in R$.
Anti-Symmetry: If $(a, b) \in R$, then $a = b$.
Transitivity: If $(a, b) \in R$ and $(b, c) \in R$, then $(a, c) \in R$.

However, I'm unsure of how to prove these for this type of problem (with functions). I would appreciate it if someone could please take the time to show and explain this.
Comment: Your anti-symmetry property is incorrect, it should be "If $(a,b) \in R$ and $(b,a) \in R$ then $a=b$". For your particular problem, given real functions $f$ and $g$ whose domain is the real line, this can be rewritten as $\left(f\in \preceq\,\land\, g\in \preceq\right)\implies f=g$, or, more familiarly looking: $$\text{If }\forall x\in \mathbb R(f(x)\leq g(x))\text{ and }\forall x\in \mathbb R(g(x)\leq f(x))\text{ then }\forall x\in \mathbb R(f(x)=g(x)).$$ Can you take the problem after it has been rewritten like this?
Comment: You say $\preceq$ is defined on $F$. So what does it mean $f(x)\preceq g(x)$? Is there another relation $\preceq$ defined on $\mathbb{R}$ which you know is a partial order?
Comment: @Mark Sorry, I made an error with typesetting. Fixed now. Is that better?
Comment: @GitGud Thanks. I will attempt the problem now, but I do not have solutions, so I have no idea what is correct and what isn't.
Answer: We don't have much to work with other than the definition of $\preceq$. In a way, this makes the problem easier since the techniques available are already presented.

Reflexivity: let $f \in \mathbb{R}^\mathbb{R}$. Now, we trivially have $f(x) = f(x)$ for all $x \in \mathbb{R}$, so $f \preceq f$.
Antisymmetry: let $f,g \in \mathbb{R}^\mathbb{R}$. Now, if $f \preceq g$ and $g \preceq f$, we have that $f(x) \leq g(x)$ and $g(x) \leq f(x)$ for all $x \in \mathbb{R}$. By antisymmetry of $\leq$, this implies $f(x) = g(x)$ for all $x$ so $f = g$.
Transitivity: suppose that $f \leq g$ and $g \leq h$, and let $x \in \mathbb{R}$. Now by definition,

$$
f(x) \leq g(x) \quad \text{ and } \quad  g(x) \leq h(x)
$$
By transitivity of $\leq$, this implies $f(x) \leq h(x)$. Since this happens for all $x \in \mathbb{R}$, we have $f \preceq h$.
As you can see from the proof, we are not using anything other than $\leq$ being an order relation.
In general, if $\leq'$ is an order on a set $X$ and $Y$ is another set, you can give an order relation on $X^Y$ via $f \preceq' g$ iff $f(x) \leq' g(x) \quad (\forall x \in X)$.
Comment: @ThePointer if $X$ and $Y$ are sets, $Y^X$ is notation for the set of functions $f: X \to Y$. In this case, $F = \mathbb{R}^\mathbb{R}$.
Comment: Thanks for the answer. Question: why $\mathbb{R}^\mathbb{R}$?
Comment: Ahh, makes sense.
Answer: It simply follows from the fact that $\leq$ is a partial order on $\mathbb{R}$. For example take a function $f \in F$. You know that for every $x \in \mathbb{R}$ we have $f(x)\leq f(x)$. Hence by your definition $f\preceq f$. And that is for every function in $F$, so the relation is reflexive.
Now, assume there are two functions $f,g \in F$ for which $f\preceq g$ and $g\preceq f$. That means, for every $x\in\mathbb{R}$ we have $f(x)\leq g(x)$ and $g(x)\leq f(x)$. From here it follows that $f(x)=g(x)$ for every $x\in\mathbb{R}$, so $f=g$. The relation is anti-symmetric. Can you now check it is transitive? It is the same idea.


How to Parse a CSV in Salesforce

Question: I have a CSV that contains fields that need to go to several Salesforce custom objects.  I've created the APEX class to handle the parsing.  However, when I call the class from a VisualForce page, I receive the message:
Collection size 3,511 exceeds maximum size of 1,000.

My APEX class is as follows (cobbled together from 2 blog posts on the topic):
public class parseCSV{
public Blob contentFile { get; set; }
public String nameFile { get; set; }
public Integer rowCount { get; set; }
public Integer colCount { get; set; }
public List<List<String>> getResults() {
 List<List<String>> parsedCSV = new List<List<String>>();
 rowCount = 0;
 colCount = 0;
 if (contentFile != null){
 String fileString = contentFile.toString();
 parsedCSV = parseCSV(fileString, false);
 rowCount = parsedCSV.size();
 for (List<String> row : parsedCSV){
 if (row.size() > colCount){
 colCount = row.size();
}
}
}
return parsedCSV;
}
public static List<List<String>> parseCSV(String contents,Boolean skipHeaders) {
List<List<String>> allFields = new List<List<String>>();
// replace instances where a double quote begins a field containing a comma
// in this case you get a double quote followed by a doubled double quote
// do this for beginning and end of a field
contents = contents.replaceAll(',"""',',"DBLQT').replaceall('""",','DBLQT",');
// now replace all remaining double quotes - we do this so that we can reconstruct
// fields with commas inside assuming they begin and end with a double quote
contents = contents.replaceAll('""','DBLQT');
// we are not attempting to handle fields with a newline inside of them
// so, split on newline to get the spreadsheet rows
List<String> lines = new List<String>();
try {
lines = contents.split('\n');
}
catch (System.ListException e) {
System.debug('Limits exceeded?' + e.getMessage());
}
Integer num = 0;
for(String line : lines) {
// check for blank CSV lines (only commas)
if (line.replaceAll(',','').trim().length() == 0) break;
List<String> fields = line.split(',');
List<String> cleanFields = new List<String>();
String compositeField;
Boolean makeCompositeField = false;
for(String field : fields) {
if (field.startsWith('"') && field.endsWith('"'))
{
            cleanFields.add(field.replaceAll('DBLQT','"'));
}
else if (field.startsWith('"')) {
makeCompositeField = true;
compositeField = field;
}
else if (field.endsWith('"')) {
compositeField += ',' + field;
cleanFields.add(compositeField.replaceAll('DBLQT','"'));
makeCompositeField = false;
}
else if (makeCompositeField) {
compositeField +=  ',' + field;
}
else {
cleanFields.add(field.replaceAll('DBLQT','"'));
}
}
allFields.add(cleanFields);
}
if (skipHeaders) allFields.remove(0);
return allFields;
}
}

How do I ensure that the above class inserts the correct columns from the CSV into the proper custom object?
Since some objects contain lookups to others, how do I ensure that data is loaded into the lookup fields first, and then into the child tables?
Finally, How do I work around the error message above?  Is something besides use of a VisualForce page suggested?
Thanks much for your help and guidance.
Comment: How did you go about testing?
Comment: Too many separate questions here ("How do I parse a CSV?", "How do I fix this particular wall of code?", "Should I use something besides a VisualForce page?", "How do I fix this particular error?") for anyone to answer comprehensively in a way that will be useful to future readers. :(
Answer: Out of interest, is there a requirement for this to be from a visualforce page? You could just use Data Loader to load CSVs, or even build a solution using the bulk API yourself.
If you do need to do it from a page, I'd suggest that rather than splitting on a new line immediately, you use substring and the indexOf to pull just one line at a time from the input string, processing them as you go.
Comment: Not in one go — is this to be a regular load or just occasional? If occasional I'd suggest doing the loads with dataloader and using the success file from the first load with a VLOOKUP to set the lookup IDs in the other tables. If it's to be a regular thing then maybe you'd be best finding an ETL tool to use?
Comment: Data Loader is probably still your best bet, you just have to script a few steps and make use of external ID columns for joins. It's basic but it is solid, I use it for production processes regularly. Others to look at: Apatar (free but I'm not really a fan of it), DreamFactory Monarch (haven't used it but it looks good), Jitterbit. Pervasive is full-featured but also crazy expensive if you just need it for a couple tables.
Comment: I was avoiding use of Data Loader since the CSV contains fields that need to be inserted into separate objects, rather than in one object.  For example, the CSV contains a CallFrom and a CallTo field which are to be loaded into the Calls object.  Those are lookups to another table, DID.  Thus, there must be entries in DID in order for CallFrom and CallTo to load successfully into the Call object.  Can DataLoader handle this use case?
Comment: Sounds like an ETL tool is the best bet as there will be routine uploads of this nature.  Aside from SSIS, do you suggest any others that do a good job in this space?
Comment: Pervasive and Flow can both work with Salesforce, but they carry a cost along with them (not sure of the details)
Comment: Talend Open Studio for Data Integration is freely available and adequate for this sort of task. http://www.talend.com/products/open-studio-di.php
Answer: I tried to process a 4MB CSV file and SalesForce returned to me a "Regext too complicated" error. I believe the issue is something similar to yours.
I fixed it using batch apex and an iterator. Further information is at this link
Answer: This example from salesforce does not handle the case where you have single quote by itself.
For example,
",",XXX,YYY

This will return 4 fields when there are only 3
The issue is at:
if (field.startsWith('"') && field.endsWith('"'))

When you have a field such as:
"

It both starts and ends with double quote but that is not the intent of the test
Change it to:
if (field.length() > 1 && field.startsWith('"') && field.endsWith('"')) {

and it will work.
The full corrected example is:
    public class parseCSV{
public Blob contentFile { get; set; }
public String nameFile { get; set; }
public Integer rowCount { get; set; }
public Integer colCount { get; set; }
public List<List<String>> getResults() {
 List<List<String>> parsedCSV = new List<List<String>>();
 rowCount = 0;
 colCount = 0;
 if (contentFile != null){
 String fileString = contentFile.toString();
 parsedCSV = parseCSV(fileString, false);
 rowCount = parsedCSV.size();
 for (List<String> row : parsedCSV){
 if (row.size() > colCount){
 colCount = row.size();
}
}
}
return parsedCSV;
}
public static List<List<String>> parseCSV(String contents,Boolean skipHeaders) {
List<List<String>> allFields = new List<List<String>>();
// replace instances where a double quote begins a field containing a comma
// in this case you get a double quote followed by a doubled double quote
// do this for beginning and end of a field
contents = contents.replaceAll(',"""',',"DBLQT').replaceall('""",','DBLQT",');
// now replace all remaining double quotes - we do this so that we can reconstruct
// fields with commas inside assuming they begin and end with a double quote
contents = contents.replaceAll('""','DBLQT');
// we are not attempting to handle fields with a newline inside of them
// so, split on newline to get the spreadsheet rows
List<String> lines = new List<String>();
try {
lines = contents.split('\n');
}
catch (System.ListException e) {
System.debug('Limits exceeded?' + e.getMessage());
}
Integer num = 0;
for(String line : lines) {
// check for blank CSV lines (only commas)
if (line.replaceAll(',','').trim().length() == 0) break;
List<String> fields = line.split(',');
List<String> cleanFields = new List<String>();
String compositeField;
Boolean makeCompositeField = false;
for(String field : fields) {
if (field.length() > 1 && field.startsWith('"') && field.endsWith('"')) {
{
            cleanFields.add(field.replaceAll('DBLQT','"'));
}
else if (field.startsWith('"')) {
makeCompositeField = true;
compositeField = field;
}
else if (field.endsWith('"')) {
compositeField += ',' + field;
cleanFields.add(compositeField.replaceAll('DBLQT','"'));
makeCompositeField = false;
}
else if (makeCompositeField) {
compositeField +=  ',' + field;
}
else {
cleanFields.add(field.replaceAll('DBLQT','"'));
}
}
allFields.add(cleanFields);
}
if (skipHeaders) allFields.remove(0);
return allFields;
}
}

Comment: The current code doesn't account for fields that are in quotes and happen to end in a comma (example "My Data,").  To fix that, all you need to do is put the if (field.endsWith('"')) block in front of the if (field.startsWith('"')) block. This code also doesn't account for blank data at the end after a comma. For instance the string 'one,two,three,' will count as 3 columns, instead of 4. To fix that, change line.spit(',') to line.split(',', -1). This code will still be problematic on large strings causing contents.split() and contents.replaceAll() to throw a 'RegEx too complicated' error.
Comment: To make this work for large files, move contents.replaceAll() to line.replaceAll() and replace contents.split('\n') with a function that uses the Matcher class which can have it's region advanced and be reset after X number of matches.
Answer: I've received that message too when I tried to store converted CSV data (over 1000 records) into a list to display on a Visualforce page.  If you are going to display imported records (which is a nice feature), you need to make sure the list you are displaying has less than 1000 elements.
To insert the records into a Salesforce object, do something like this:
I defined a class called "Util" with a static method called "parseCSV" (see below):
public blob importedCSV {get;set;}

list<list<string>> parsedCSV = Util.parseCSV(importedCSV.toString(),true);
list<Custom_Object__c> cObjsToInsert = new list<Custom_Object__c>();
for(list<string> line:parsedCSV){
    Custom_Object__c c = new Custom_Object__c();
    c.Name=line[0];
    c.String_Field__c=line[2];
    c.Currency_Field__c=decimal.valueof(line[3]);
    if(line.size()>4)
        c.Description__c=line[4];
    cObjsToInsert.add(c);
}
insert cObjsToInsert;

Usually, when I process a CSV and insert the data into Salesforce objects, I don't run into size-errors.  I've made Visualforce pages that import a CSV, processes the CSV in some fancy way, and then export the processed data back into a CSV,Excel, or text (fixed width) format.  I'll load the CSV on a Visualforce page and save the data as an attachment on a dummy record.  Then, I'll direct the user to another Visualforce page that grabs the attachment, processes the data, and outputs to the desired format.
I can't remember where I got this code, but it works like a charm.  I wrote test code for the parseCSV method too.
public static list<list<String>> parseCSV(String contents,Boolean skipHeaders) {
    list<list<String>> allFields = new List<List<String>>();

    // replace instances where a double quote begins a field containing a comma
    // in this case you get a double quote followed by a doubled double quote
    // do this for beginning and end of a field
    contents = contents.replaceAll(',"""',',"DBLQT').replaceall('""",','DBLQT",');
    // now replace all remaining double quotes - we do this so that we can reconstruct
    // fields with commas inside assuming they begin and end with a double quote
    contents = contents.replaceAll('""','DBLQT');
    // we are not attempting to handle fields with a newline inside of them
    // so, split on newline to get the spreadsheet rows
    list<String> lines=new list<string>();
    try {
        lines = contents.split('\n');
    } catch (System.ListException e) {
        System.debug('Limits exceeded?  '+e.getMessage());
    }
    integer num=0;
    for(string line :lines) {
        // check for blank CSV lines (only commas)
        if(line.replaceAll(',','').trim().length()==0)
            break;
        list<String> fields=line.split(',');
        list<String> cleanFields=new list<String>();
        string compositeField;
        boolean makeCompositeField=false;
        for(string field : fields) {
            if(field.startsWith('"') && field.endsWith('"')) {
                cleanFields.add(field.replaceAll('DBLQT','"').removeStart('"').removeEnd('"'));
            } else if(field.startsWith('"')) {
                makeCompositeField = true;
                compositeField = field;
            } else if(field.endsWith('"')) {
                compositeField += ',' + field;
                cleanFields.add(compositeField.replaceAll('DBLQT','"').removeStart('"').removeEnd('"'));
                makeCompositeField = false;
            } else if(makeCompositeField) {
                compositeField +=  ',' + field;
            } else{
                cleanFields.add(field.replaceAll('DBLQT','"').removeStart('"').removeEnd('"'));
            }
        }
        allFields.add(cleanFields);
    }
    if(skipHeaders)
        allFields.remove(0);
    return allFields;
}//END ParseCSV

@isTest
private static void testParseCSV(){
    string contents='Field1,Field2,Field3\n1,,Smith\n2,Fred,O\'Connor\n3,Destiny,"Awaits, DDS"\n\n';
    list<list<string>> parsedCSV=Util.parseCSV(contents,true);
    //line 1
    system.assertEquals('1',parsedCSV[0][0]);
    system.assertEquals('',parsedCSV[0][1]);
    system.assertEquals('Smith',parsedCSV[0][2]);
    //line 2
    system.assertEquals('2',parsedCSV[1][0]);
    system.assertEquals('Fred',parsedCSV[1][1]);
    system.assertEquals('O\'Connor',parsedCSV[1][2]);
    //line 3
    system.assertEquals('3',parsedCSV[2][0]);
    system.assertEquals('Destiny',parsedCSV[2][1]);
    system.assertEquals('Awaits, DDS',parsedCSV[2][2]);
}//END

I placed this code in a class called "Util".  If you have a blob, first convert it to a string, and call the method.   e.g.
blob b=inputBlob;
list<list<string>> parsedCSV = Util.parseCSV(b.toString(),true);

How to Parse a CSV in Salesforce


Olympic sports with mixed team?

Question: What are, if they exist, Olympic sports (both winter and summers Games) where there are team composed by men and women?
Comment: Are you considering Paralympics also?
Comment: [Mixed doubles](https://en.wikipedia.org/wiki/Tennis_at_the_Summer_Olympics#Events) in tennis.
Comment: There is no mixed-sex shooting at the Summer Olympics although there is a proposal for 3 mixed pair events for Tokyo.
Comment: The question was about existing sport, anyway it is better to add references/links
Answer: Wikipedia Category:Mixed sports at the Olympics contains:

Mixed doubles badminton at the Summer Olympics
Equestrian at the Summer Olympics
Ice dancing at the Winter Olympics
Pair skating at the Winter Olympics
Mixed-sex shooting at the Summer Olympics
Mixed doubles tennis at the Summer Olympics

A further event, Mixed Doubles Curling is planned for the Winter Olympics 2018 in PyeongChang

Wikipedia article Women at the Olympics mentions that:

While team mixed gendered sports are not competed at the Olympics, some mixed gendered events are included. They include equestrian sports, shooting and sailing where men and women compete against each other. In shooting and sailing, women were originally only allowed to competed in mixed gendered events. Single gender events for these sports were not added until a later date.
Answer: The events of the Equestrian sports in the summer Olympics are always with mixed teams.
In the Show Jumping, Dressage and Eventing disciplines men and women compete on equal terms.  This is both in the individual and team events.
The sport of Equestrianism has always been mixed, not just in the Olympic games.

Olympic sports with mixed team?


json-server returns 404 for specific URI

Question: I use json-server and my codes are like below.
db.json
{
  "en-boards": [
    {
      "id": 0,
      "name": "News",
      "uri": "/board/en-news"
    },
    {
      "id": 1,
      "name": "Politics",
      "uri": "/board/en-politics"
    }
  ],
  "board-en-news-threads-latest": [
    {
      "id": 0,
      "poster": null,
      "zero-post": "This is a thread 0.",
      "attachment-path": null,
      "post-number": 100,
      "utc-timestamp": "2022-01-01T00:00:00"
    },
    {
      "id": 1,
      "poster": null,
      "zero-post": "This is a thread 1.",
      "attachment-path": null,
      "post-number": 100,
      "utc-timestamp": "2022-01-01T00:00:00"
    }
  ]
}

routes.json
{
  "/api/*": "/$1",
  "/api/en-boards": "en-boards",
  "/api/board/en-news/threads/latest": "board-en-news-threads-latest"
}

I run json-server --watch db.json --routes routes.json --middlewares middleware.js --port 4000
When I run curl -i http://localhost:4000/api/en-boards, I get 200 OK, but when I run curl -i http://localhost:4000/en-news/threads/latest, I get 404 Not Found. wget and React app behave in the same way.
Environment

MacOS 13.0
Node.js 18.11.0
json-server 0.17.0
curl 7.84.0
wget 1.21.3
React 18.2.0

Thank you for read. Any help would be appreciated.
Answer: Have faced something similar before...
In my case, I needed something like base URLs (1) and endpoint paths (2) like the below -

{
"/api/board/*": "/$1",
"/api/*": "/$1",
"/en-boards": "/en-boards",
"/en-news/threads/latest": "/board-en-news-threads-latest"}

This would cater to the following endpoints being hit
http://localhost:3000/api/board/en-news/threads/latest
http://localhost:3000/api/en-news/threads/latest
http://localhost:3000/en-news/threads/latest

http://localhost:3000/api/board/en-boards
http://localhost:3000/api/en-boards
http://localhost:3000/en-boards

Note: for /api/board/en-news/threads/latest to route to /board-en-news-threads-latest, the order for the base URLs needs to be maintained (i.e.)
"/api/board/*": "/$1",
"/api/*": "/$1",

Hope this helps :)


What is the suitable IP address for DNS?

Question: i am configure a new Windows Server in a new network however i do not know what should i put for my DNS server address. Here is my network configuration:
IP: 10.0.0.7
subnet mask: 255.255.255.0
Gateway: 10.0.0.5
Please guide me on this. Thank you and appreciate for the help
Comment: You should use the IP address of the DNS server. Kind of obvious if you stop to think about it.
Answer: Given it's an internal address then you'll either need to use an existing internal DNS server or if your gateway is setup to just NAT everything (which I doubt it is) then you could try someone like Google's DNS servers at 184.108.40.206 or 126.96.36.199.
Answer: It depends on what you have configured the server to be. If its an Active Directory Server with integrated DNS Zone, Microsoft recommends you add the loopback address (127.0.0.1) as the primary DNS then another DNS Server in the Domain as the secondary. You could then set a forwarder in your internal DNS server to an external DNS Server for internet name resolution.
If you just want the server to be able to browse the internet then all you need is to follow Choppers advice
Answer: Best option: ask you network administrator, i.e. the one that gave you the IP 10.0.0.7.
Other option: ask your ISP for the DNS servers they suggest.
Where possible, I personally use 184.108.40.206 18.104.22.168
Answer: DNS
You can use any DNS you want. For example I always use 126.96.36.199. But the best option is to use:

The DNS recommended by your ISP (probably the DNS that other machines in your network are using now)
If not sure use the one that you have in your machine configured (the one you are using to read this)

IP Address
You are using a Private class A that allows you to use 16Million of IPs (exactly 16.777.216). Maybe you should use a class C. Usually people use class C for small networks.
Here you have an example of the different types of Private Class Addresses:

Private Class C: IP: 192.168.0.7
subnet mask: 255.255.255.0 Gateway:
192.168.0.5
Private Class B: IP: 172.16.0.7
subnet mask: 255.255.0.0 Gateway:
172.16.0.5
Private Class A: IP: 10.0.0.7 subnet
mask: 255.0.0.0 Gateway: 10.0.0.5

You have more info here :
http://en.wikipedia.org/wiki/IP_address (Check IPv4 private addresses section)
EDIT NOTES:

I am assuming that you want your host to be x.x.x.7 and your gateway will be x.x.x.5
Be careful that all the machines in your network must have same subnet mask.
Comment: You may want to finish reading the entire article to which you've linked... [Classes are obsolete terminology](http://en.wikipedia.org/wiki/Classful_network#The_replacement_of_classes), please read up on [CIDR](http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) as this supplanted classes many years ago..
Comment: Explaining something incorrectly is not beneficial to anyone. Please have a [search for "classless"](http://serverfault.com/search?q=classless) to get a feel for how strongly this community feels about the subject.
Comment: @jscott. Thanks! I know. But I thought explaining this in depth will cause more trouble than benefit to @asley


How are different sport climbing disciplines combined?

Question: Last year the IOC decided to include sport climbing in the 2020 Olympics. Sport climbing covers three distinct disciplines: lead, boulder and speed.
The 2020 event will be a combination of those three disciplines. Is there a standard way to combine the performances in those three disciplines? In each of them performances are measured in entirely different ways: highest hold reached, number of problems climbed and time. How will those three be combined to a meaningful result?
Comment: They seem surprisingly reticent about any remotely useful information, beyond the general information about each discipline available anyway and the basic entry/medal count. Quite possibly the score system has not even reached an appropriate draft stage for public release or for athlete consultation, but I am expecting something similar to the methods for pentathlon and heptathlon and decathlon.
Comment: The answer below covered the technical side of this question. However, from a more "opinion" side, it's clear from the IOC's decision for the 2024 games — lead and bouldering are together, speed is separate — that it doesn't make sense to combine the three for any meaningful result. Speed climbing is a very, very different beast.
Answer: The three results will be combined by considering only the rank in each discipline and multiplying all results, see here. The lowest combined rank wins.
There will be a qualification and then finals. 20 athletes of each sex (20 men and 20 women) will be in the qualification as far as I know and the best 6 will go to the finals.
Speed will be first, than bouldering and finally lead.
How does multiplying the ranks work and what does it mean?
To give an example, consider a speed climber who climbs reaches the top in speed with the lowest time (1st rank in speed) but who is not among the top group in either lead (say 15) or bouldering (say 18). He/she will have a combined score of 1*15*18=270.
This speed climber will be better than an mediocre guy who gets the 10th place in each discipline (10*10*10=1000). This shows that specialists are favored compared to all-discipline people (if one would look at the average rank, the all-discipline guy would win with 10 compared to (1+15+18)/3=11.33 ).
However, the overall winner will probably be someone who is skilled in bouldering and lead (speed needs a very different skill set compared to bouldering/lead). To see that, think of someone is last in speed, but first in bouldering and third in lead. This guy will have a score of just 1*3*20=60, much lower than the above described speed climber (even if he/she would be say 10th in bouldering or lead).


How the effects(Q'*y) in lm.fit calculated?

Question: In R, lm.fit return an effects variable which equals to Q'Y or Rb (X=Q*R). However, I am confused by the dimension of this variable. In my case, dim(X)=6*2, dim(Q)=6*2, dim(R)=2*2, dim(Y)=6*10, so the dim(Q'*Y) should be 2*10, whereas lm.fit gave a 6*10 matrix. The first two rows of effects are identical to Q'*Y, but I have no idea how the remaining rows were calculated.
Answer: This is covered in Section 4.4.1 in the White Book (Statistical Models in S, Chambers & Hastie (eds), 1991; CH henceforth). But basically, you are confusing the matrix $\mathbf{Q}$ with its submatrix $\mathbf{Q}_1$ following the notation in CH.

To elaborate, $\mathbf{Q}$ is the orthonormal $N \times N$ matrix such that $\mathbf{Q}'.\mathbf{Q} = \boldsymbol{\iota}$, where $\boldsymbol{\iota}$ is the identity matrix. It is chosen so that
$$
\mathbf{Q}'.\mathbf{X} = \begin{bmatrix}\mathbf{R} \\ \mathbf{O}\end{bmatrix}
$$
where, $\mathbf{R}$ is an upper triangular matrix.
All the relations you state, are only true of the submatrix $\mathbf{Q}_1$ of $\mathbf{Q}$
$$
\mathbf{X} = \mathbf{Q}_1.\mathbf{R}
$$
where the columns of $\mathbf{Q}_1$ span the columns of $\mathbf{X}$. This means that $\mathbf{Q}_1$ is an $N \times K$ submatrix of $\mathbf{Q}$ ($K$ is the total number of regressors).
Next, on page 133, CH clearly define effects (vector $\boldsymbol{C}$ below), and also point out the elements that correspond to $\mathbf{Q}_1$:
$$
\begin{align}
\boldsymbol{C} &= \begin{bmatrix}\boldsymbol{C}_1 \\ \boldsymbol{C}_2 \end{bmatrix} \\
&= \mathbf{Q}'\boldsymbol{Y}
\end{align}
$$
where $\boldsymbol{C}_1$ is of length $K$, and $\boldsymbol{C}_2$ is of length $N-K$. These are what you get as named elements of lm.fit.obj$effects.
And, as you have rightly figured out, the least squares solution, $\widehat{\boldsymbol{\beta}}$ is any vector that solves:
$$
\mathbf{R}\widehat{\boldsymbol{\beta}} = \boldsymbol{C}_1 = \mathbf{Q}'_1\boldsymbol{Y}
$$

Here is an example:
# generate some data
set.seed(1)
mX = matrix(rnorm(6*2), nrow = 6, ncol = 2)
vBeta = rnorm(2)
vY = rnorm(6, mX %*% vBeta)

# fit the model
lmA = lm.fit(x = mX, y = vY)

# Q1'.Y = C1
t(qr.Q(lmA$qr)) %*% vY

# full effects vector
lmA$effects

Comment: Thanks a lot for your information. A further question about the value of C2 in the matrix of Q'*y remained. If matrix Q was defined in NxN format, then C2 should be 0, however this contradict to the output of lm.fit. Then what's the actual value of C2?
Comment: @user64629 Why should $\boldsymbol{C}_2$ be $\boldsymbol{0}$? $\boldsymbol{C}_1 = \mathbf{Q}_1\boldsymbol{Y}$ and $\boldsymbol{C}_2 = \mathbf{Q}_2\boldsymbol{Y}$.
Comment: @ fg nu I corrected my assersion, as shown below
\begin{align}
Q' \cdot Y &= Q' \cdot (X \cdot \widehat{\boldsymbol{\beta}} + \boldsymbol{\epsilon})\\ &=\begin{bmatrix} R\\ 0\\ \end{bmatrix} \cdot \widehat{\boldsymbol{\beta}} + Q' \cdot \boldsymbol{\epsilon}\\ &= \begin{bmatrix} R \cdot \widehat{\boldsymbol{\beta}}\\ 0\\ \end{bmatrix} + \begin{bmatrix} D1\\ D2\\ \end{bmatrix}
\end{align} Since the top rows of effects matrix are identical to $R \cdot \widehat{\boldsymbol{\beta}} $, did this means two points? 1. $ D1 = 0 $
2. $ C2 = D2 = $ submatrix of $ Q'\cdot \boldsymbol{\epsilon} $
Comment: @divy Those computations are correct.


Get ViewModel for products and categories

Question: I am developing an ecommerce website and I use three tables for the management of products and categories.
I would need one or more linq queries with entity framework to get the list of mother categories, daughters and the list of related products.
There is a Category table, like:
Id
NameCat
ParentId
There is a products table, like:
Id
Name
Live
Info, etc.
There is a product-category table, like:
Id
categoryId
productId
sortOrder
This is my viewmodel
public class CatViewModel
    {
        public int IdCat { get; set; }
        public string NameCat { get; set; }
        public int ParentId { get; set; }
        public List<CatViewModel> Children { get; set; }
        public List<Product> Products { get; set; }

    }

Clearly each parent category must have the complete list of products of all its child categories
Answer: You are on the right track. In the method that is supposed to return view with this CatViewModel you can do this:
var mainObject = Context.Category.FirstOrDefault(x => x.IdCat == "YOUR PARAMETER OF ID HERE");
var viewModel = new CatViewModel
{
   IdCat = mainObject.IdCat,
   NameCat = mainObject.NameCat,
   ParentId = mainObject.ParentId,
   Children = Context.Children(query here).ToList(),
   Products = Context.Products(query here).ToList()
};

Solution 2:
You can first create:
var listChildern = Context.Children(query here).ToList();
var listProducts = Context.Products(query here).ToList();

Then return your viewModel which you fill just like this:
var mainObject = Context.Category.FirstOrDefault(x => x.IdCat == "YOUR PARAMETER OF ID HERE");
var viewModel = new CatViewModel
{
   IdCat = mainObject.IdCat,
   NameCat = mainObject.NameCat,
   ParentId = mainObject.ParentId,
   Children = listChildern ,
   Products = listProducts
};

Comment: I have three tables.
I need a CatViewModel list type model, I tried this:
var cats = db.Cats.AsEnumerable() // <-- Force full execution (loading) of the above
                                .Where(e => e.ParentId == null) // <-- then apply the root filter
                                .ToList();
                var myList = cats.Select(c => new CatViewModel
                {
                    IdCat = c.IdCat,
                    NameCat = c.Nomecat_it,
                    Children = c.Children,
                    Products = ???
                }).ToList();
Answer: I have three tables.
I need a CatViewModel list type model, I tried this:
var cats = db.Cats.AsEnumerable() // <-- Force full execution (loading) of the above
.Where(e => e.ParentId == null) // <-- then apply the root filter
.ToList();

var myList = cats.Select(c => new CatViewModel {
 IdCat = c.IdCat,
NameCat = c.Nomecat,
Children = c.Children,
Products = ???
}).ToList();


Redact XML using XSLT

Question: Wondering if anyone can help with an XSLT issue I am facing.
I am trying to create an xslt script which will take take as input an xml document and change the values of several fields to "xxxx" I have managed to get this part working however I would now only like this to run if one field in the input xml is of a specific value (e.g. if username is jbond)
I like to have this condition within my XSLT if possible however I am having difficulty.
My current XML, XSLT, Output and expected outputs are as follows
XML:
     <?xml version="1.0"?>
        <?xml-stylesheet type="text/xsl"?>
        <rootDoc>
        <user>test</user>
        <tel>12345</tel>
        <zip>abcd</zip>
</rootDoc>

XSLT:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions" >
    <xsl:output method="xml" omit-xml-declaration="no" indent="yes"/>

 <xsl:template match="node()|@*">
<xsl:if test="user = 'test'">
      <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
</xsl:if>
  </xsl:template>

<xsl:template match="tel/text()">XXXX</xsl:template>
<xsl:template match="zip/text()">XXXX</xsl:template>

</xsl:stylesheet>

Output:
<?xml version="1.0" encoding="UTF-8"?>
<rootDoc/>

Expected:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl"?><rootDoc>
   <user>test</user>
   <tel>XXXX</tel>
   <zip>XXXX</zip>
</rootDoc>

Answer: Putting the test into the identity transformation template does not make much sense, if you want to perform certain changes when a condition holds then the templates
<xsl:template match="tel/text()">XXXX</xsl:template>
<xsl:template match="zip/text()">XXXX</xsl:template>

should be changed to
<xsl:template match="rootDoc[user = 'test']/tel/text()">XXXX</xsl:template>
<xsl:template match="rootDoc[user = 'test']/zip/text()">XXXX</xsl:template>

which could be joined into
<xsl:template match="rootDoc[user = 'test']/tel/text() | rootDoc[user = 'test']/zip/text()">XXXX</xsl:template>

With a single condition, assuming it is the rootNode, you can use
<xsl:template match="/rootNode[not(user = 'test')]">
  <xsl:copy-of select="."/>
</xsl:template>

then the other cases are handled by the identity transformation and the specialized templates.
Comment: Thanks for your reply, Would you know if it's possible to have a more generic solution which will only run the condition once rather than for each node. in the final document there are several fields we wish to sanitize and for this reason we are trying to reduce the number of repetitive conditions
Comment: If you know you don't want to change anything in case the condition does not hold then you can use a template for the root element which simply copies everything instead of applying templates.
Answer: If you leave the identity transform alone, but add specific matches, XSLT will automagically find the closest match. You can customize the select="" or add more templates as necessary. hth
    <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions" >
    <xsl:output method="xml" omit-xml-declaration="no" indent="yes"/>

 <xsl:template match="node()|@*">
     <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

<xsl:template match="tel"><tel>XXXX</tel></xsl:template>
<xsl:template match="zip"><zip>XXXX</zip></xsl:template>

</xsl:stylesheet>

Comment:
test
XXXX
XXXX

Redact XML using XSLT


Finite dimensional central division algebras over a finite extension of $\mathbb{F}_q(T)$

Question: Over number fields, finite dimensional central division algebras are always cyclic algebras. So the construction of cyclic algebras is a nice recipe to create algebras, which exhausts all finite dimensional central division algebras over number fields.
Is there a similar recipe for finite dimensional central division algebras over functions fields (by which I mean finite extensions of $\mathbb{F}_q(T)$)?
If not, and this is a somewhat different question, what are the some general families of such algebras, and are they known not to exhaust everything? Further, what if we restrict only to $4$-dimensional such algebras?
Comment: All central simple algebras over a global field are cyclic.
Comment: @KCd: Do you have a reference for that?
Comment: I posted an answer with more information.
Answer: All central simple algebras over a global field are cyclic algebras. In Milne's class field theory notes he treats the case of number fields in section 2 of chapter VIII as a consequence of a corollary on the existence of a cyclic extension of a number field with finitely many specified local degrees. That same existence result on cyclic extensions is handled for all global fields as Theorem 5 in Chapter 10 (Grunwald's Theorem) of the Artin-Tate notes on class field theory.


No collapsable snippets on Magento SE?

Question: On other SE pages you can use collapsible code snippets like in this post, but it doesn't work on Magento SE.
Using this ...
<code><!-- begin snippet: js hide: true -->

<!-- language: lang-js -->

alert(2+ 40);

<!-- end snippet -->
</code>
Just outputs ...

alert(2+ 40);

Is there a special reason for this? I want to write a "longer" question where i would really like to use this.
Answer: As you already mentioned in your question, this question is about Code Snippet. However, code snippet is not enabled for MSE.
The feature what we have available right now is Code Blocks for which SE uses Google Code Prettify internally. So the expand/collapse-able feature would be possible only if the Code Prettify has such facility, which I don't think exist now.

Code Snippet != Code Blocks

I am not sure when and how we will get the code snippet feature here. I think this code snippet feature is only available in the site StackOverflow. So the chance to get this feature for MSE is very rare (since there exists pretty old SE sites such as SuperUser which do not have code snippet feature yet.)
Refer this answer: https://meta.stackoverflow.com/a/354002/2869218

No collapsable snippets on Magento SE?


CollectionView preferredLayoutAttributesFItting function

Question: I have an override function for my collection view:
    // Sets a the width as the screen width and has a dynamic height that changes depending on how much is inside the cell.
override func preferredLayoutAttributesFitting(_ layoutAttributes: UICollectionViewLayoutAttributes) -> UICollectionViewLayoutAttributes {
    let targetSize = CGSize(width: layoutAttributes.frame.width, height: 40)
    layoutAttributes.frame.size = contentView.systemLayoutSizeFitting(targetSize, withHorizontalFittingPriority: .required, verticalFittingPriority: .fittingSizeLevel)
    print("Layout has been changed.")
    return layoutAttributes
}

This function makes the cell heights dynamic depending on the height of the contents inside the cell while keeping the width a static width of the phone screen. This function works exactly how I want it to work.
The problem that I am having is this function is only called when the view loads, and I want it to also run when I call collectionView.reloadData(). So when I call reloadData() the cell goes off of its estimatedItemSize instead of its size from the function above preferredLayoutAttributes.
Does anyone know how I can allow this function to run when I call collectionView.reloadData()?
Answer: This works for me:
extension UICollectionView{
func reloadData2(_ completion: @escaping () -> Void) {
        UIView.animate(withDuration: 0, animations: {
            self.reloadData()
        }, completion: { _ in
            completion()
        })
    }

}
Comment: Unfortunately that didn't work for me :(. Function is still not being called. Am I calling it correctly? `self.collectionView.reloadData2 {}`
Comment: Call your function inside the curly braces
…reloadData2 {Yourfunction()}
Comment: In addition, if you can change the collectionview to a uitableview, you can use the autodimension.
https://stackoverflow.com/questions/30494702/dynamic-height-issue-for-uitableview-cells-swift


Obter valor da semana que vai começar um mês

Question: Quero obter o valor da semana,em que vai comecar um determinado mês. Funciona para o mês actual, mas quero para os restantes meses. Este é o codigo que utilizei para calcular, em que day é o dia de hoje.

$day = date("d");
$numjour = date("N",time()-(($day-1)*3600*24));

Comment: Com valor da semana você quis dizer o dia da semana que vai ser o dia 1º do certo mês? Por exemplo, o atual mês de novembro teve o dia 1 num sábado. Você quer que ele te retorne "Sábado"?
Comment: Mas se você setar `$day` para o dia que começa o mês ele te retornara a semana.
Comment: Por exemplo para hoje, o resultado é 6 que é sexta feira.
Comment: Quero que me da o resultado para outros meses, alterando um codigo para uma variavel $mes.
Answer: Você pode usar a função date() para este objetivo. Ao passar a opção 'w' ele retorna o número da semana de um certo dia.
Primeiro você transforma a data em timestamp, com a função strtotime();. O formato da data vai ser  $mes . "/01/" . date('Y'), ou seja, o primeiro dia do mês $mes do ano atual. Observe que a data está no formato americano "mes/dia/ano".
Então você passa esse timestamp para a função date() com a opção 'w' que retorna o número do dia da semana correspondente àquela data.
function primeiroDiaDoMes($mes) {
    $dia =  (int) date("w", strtotime($mes . "/01/" . date('Y')));
    return $dia;
}
echo primeiroDiaDoMes("11");

Exemplo: http://codepad.org/QJvxuuMQ
Vai imprimir '6', pois o mês de novembro começou em um sábado, que é representado pelo número 6. Domingo é representado em '0'.


Extrair texto de imagem sem usar Tesseract

Question: Tô querendo extrair um valor de uma imagem com um script Python, porém, a principal solução sugerida é o Tesseract, que infelizmente não consigo implementar, já tentei de diversas formas. Gostaria então de saber se existe outra ferramenta mais simples pra executar essa tarefa com Python.
Answer: O termo para isto provavelmente é OCR, o que pode ajudar a encontrar outras soluções (existem outros termos e até soluções), mas provavelmente você esqueceu de instalar o PIL ou não soube como instalar o pytesser
O pytesser pode ser instalado manualmente:

https://code.google.com/archive/p/pytesser/

Ou via pip:
pip install pytesseract

Mas antes de instalar ele você deve instalar o:

http://www.pythonware.com/products/pil/

E um exemplo da breve documentação/readme:
from pytesser import *

image = Image.open('fnord.tif') # Abre uma imagem usando PIL

print image_to_string(image)

Ou
from pytesser import *

print image_file_to_string('fnord.tif')

Claro que essa lib esta sem atualização desde 2007, logo penso que deve ter uma série de problemas com Python3 (ou nem ser compatível, não pude testar), então a solução que creio que a maioria irá lhe informar é usar o opencv, no caso existe uma pequena lib pronta que pode instalar via PIP:

https://pypi.org/project/simple-opencv-ocr/

Ela foi inspirada nas soluções propostas em https://stackoverflow.com/q/9413216/1518921, não estou certo se a lib em si reconhece apenas números, como foi proposto na pergunta original, mas se o intuito é adaptar ou até aprender esta ai alguns exemplos que pode estudar.
Comment: Obrigado, pela resposta, dei uma boa estudada nesses e outros exemplos. Percebi que terei de utilizar o Tesseract mesmo. Até o momento não encontrei uma solução mais simples.


Samba replacement for Linux and Windows

Question: I have a lot of machines (Windows and Linux) and one samba server.
I want to improve samba performance. How I should do this? I need a distributed file system, preferably Open Source. :)
I use glusterFS and it's very good, because I need a solution that works on Windows too, not only Unix.
Comment: Seconding SvenW's commentary - explain your problem more fully. \
Answer: You don't us tell anything about your problems. Please enhance your question with the following:

Why do you want to replace Samba?
What are your requirements?
What is your current hardware and your budget?
Do you want your solution free as in beer or free as in speech?
AFAIK GlusterFS doesn't work on Windows, but need a Samba share export. What do you mean?

Anyway, something to look into:

Clustered Samba
Faster servers
DFS shares with Samba
Comment: So, esio, what have you done to prove that the problem is the Samba software itself, as opposed to an IO bottleneck on your storage controller (for example) ?
Comment: I want to replace Samba because I read and save large amount of data and it's slowly.
I require very fast and scallable solution for Windows and Linux.
Answer:
because I need solution that works on Windows too

Either you've not explained your problem very well or you don't understand the difference between the file sharing protocol and the disk filesystem.
In principle you could set up a cluster of Linux boxes using glusterFS for the storage, Samba for file service and a load balancer between the cluster and the MSWindows clients. If you decide to go down this route, your life will be a lot simpler if you use CTDB. Or you could use Samba's own clustering solution (although I'm not sure hoow stable this is).
If, on the other hand, you require to have MSWindows boxes acting as servers then certainly this isn't going to work - in which case have a look at AFS.
Comment: Windows as an AFS server is unsupported. http://docs.openafs.org/ReleaseNotesWindows/ch03s11.html

Samba replacement for Linux and Windows


Sampling distribution of sample means - some basic questions that I have not found an answer to.

Question: Okay, Assume each sample has mean $x$ and standard deviation $y$ then the mean of the sample means would be $x$ and standard deviation of the sample means would be $\frac{y}{n}$ what is n here? Is it the number of samples we have, or is it the number of elements of the samples we have? I'm confused.
And also if I have a population with normal distribution I with say mean $x$ and deviation $y$ I immediately conclude that samples which are identically and independently distributed have the same mean and deviation, how come? And I just realized this is not the case for normal populations only, it is also the case for whatever distribution the population has.
Answer: Suppose you have a sample $X_1, X_2, \dots, X_n,$ having $n$ observations,
randomly sampled from a population with mean $\mu,$ variance $\sigma^2,$
and standard deviation $\sigma.$
Random sampling means that the
expectation (or mean) of each observation $X_i$ is $E(X_i) = \mu,$
that the variance of each observation is $Var(X_i) = \sigma^2,$
that the SD of each observation is $SD(X_i) = \sigma,$ and that
the random variables $X_i$ are independent of each other.
Then, one can prove, that the sample mean $\bar X = \frac{1}{n}\sum_{i=1}^n X_i,$ has $E(\bar X) = \mu.$ (We say that the sample mean
$\bar X$ is an unbiased estimator of the population $\mu.$)
Also, $Var(\bar X) = \sigma^2/n$
and $SD(\bar X) = \sqrt{Var(\bar X)} = \sigma/\sqrt{n}.$
Moreover, the sample variance
$S_X^2 = \frac{1}{n-1}\sum_{i=1}^n(X_i - \bar X)^2$ has $E(S_X^2) = \sigma^2.$
Please be sure to distinguish between observations within a sample and samples,
between population means and sample means, and between population variances
and sample variances.
Also, it is not a good idea to use symbols such as $x$ and $y$ for means and variances; I assume you did that to avoid problems printing
$\mu$ and $\sigma^2$; that can be done by typing $\mu$ and $\sigma^2$, respectively.


What is the relation between maximal tori of $U(n)$ and $SU(n)$?

Question: What is the relation between maximal tori of $U(n)$ and $SU(n)$?
Maximal torus is defined as the maximal abelian subgroup that is connected. For $U(n)$ it is the following:

The article on Wikipedia then adds that "A maximal torus in the special unitary group $SU(n) \subset U(n)$ is just the intersection of $T$ and $SU(n)$ which is a torus of dimension $n − 1$."
My questions are:

Why is the torus of $SU(n)$ intersection of $T$ of $U(n)$ with $SU(n)$?

If we take $n = 2$, does this simply mean that $T$ of $U(2)$ is of dimension $1$, so it is somehow homeomorphic to a circle..?

What does the notation "$e^{i \theta_1}$... mean? (I am sorry I am not very familiar with this, I study mostly different parts of Maths).

Thank you very much!
Comment: $e^z$ is the exponential function which can be defined for complex $z$ as $\sum_{n=0}^{\infty}z^n/n!$. In this particular case, when $z=i\theta$ and $\theta\in\mathbb{R}$ it is just $e^{i\theta}=\cos\theta+i\sin\theta$.
Comment: You should be careful with the articles: There is (in general) not "the" maximal torus. The diagonal matrices are "a" maximal torus, i.e. just one, and there are many others. For example, any conjugate of a maximal torus is again another maximal torus. For such other maximal tori, you could ask the same question.
Answer: In order:

Well $T\cap SU(n)$ is also a torus since it is still abelian and connected so you just need to see that it is maximal. It shouldn't be too hard to convince yourself that a matrix which commutes with all the (determinant 1) diagonal matrices must be diagonal itself (so in $T$) and since it is supposed to be in $SU(n)$ it must be in our torus $T\cap SU(n)$ so it is maximal.

Yes. Indeed here it is exactly given as the set of matrices:
$$ T \cap SU(2) = \left\{\left.\begin{pmatrix} e^{i\theta} & 0 \\ 0 & e^{-i\theta}
\end{pmatrix}\right|\theta \in \mathbb{R}\right\}$$

This is clearly isomorphic to the unit circle in the complex plane: $\left\{\left.e^{i\theta} \in \mathbb{C} \right|\theta \in \mathbb{R}\right\}$. We could replace $\theta \in \mathbb{R}$ by $\theta \in [0,2\pi)$ of course.

$e^{i\theta}$ is just a complex number (of unit length) written in polar form: $e^{i\theta} = \cos\theta + i \sin\theta$


Add CSS class to a flagged node

Question: I want to add CSS classes to flagged nodes in order to theme them.
Found the documentation How to add CSS classes to nodes, based on flags set, but this solution is for D6.
Can someone help me to figure out how to make it work with D7?
Comment: Have you tried that code in your own `yourtemplate_preprocess_node()`?
Answer: Try this in your template.php of your theme. Change MYTHEME to the name of your theme and bookmark to the machine name of your flag. And change node-flag-bookmark to the css class you want to be added to your node.
<code>/**
 * Implements template_preprocess_node().
 */
function MYTHEME_preprocess_node(&$variables) {
  $flag = flag_get_flag('bookmark'); // Machine name of your flag.
  if ($flag && $flag->is_flagged($variables['node']->nid)) {
    $variables['classes_array'][] = 'node-flag-bookmark';
  }
}
</code>
Comment: Thanks, it worked! Class are added when you refresh the page. It is possible to add class without a page refresh?

Add CSS class to a flagged node


how to plot real time line graph from dbn classifier train

Question: I'm working on a face recognition system using the DBN algorithm. when training data, the system will produce an error according to n-epoch. I want to make an error graph plot based on n-epoch
Training Code
    classifier = SupervisedDBNClassification (hidden_layers_structure=[200, 200],
    learning_rate_rbm=0.0001,
    learning_rate=0.01,
    n_epochs_rbm=10,
    n_iter_backprop=100,
    batch_size=32,
    activation_function='relu',
    dropout_p = 0.2)

if we run that code, it will produce
    >> Epoch 84 finished    ANN training loss 0.681700
    >> Epoch 85 finished    ANN training loss 0.682314
    >> Epoch 86 finished    ANN training loss 0.680272
    >> Epoch 87 finished    ANN training loss 0.680542

Comment: so how i can plot it? This is the link where I got the code https://github.com/albertbup/deep-belief-network
Answer: Use history = classifier.fit(x_train, y_train)
# list all data in history
print(history.history.keys())

You can access all the information like below to plot a line graph
history.history['acc'])
history.history['val_acc']
history.history['loss'])
history.history['val_loss']

For futher, look at this link


Mediation analysis with an ordinal mediator

Question: I have to do mediation analysis for my research and my IV and DV are both continuous and mediator variable is ordinal (time spent on Instagram where categories are:

less that 30 minutes a day
30 mins to 3h a day
3h to 6h a day
more than 6h

I labeled this mediator as ordinal since measure differences in the categories aren't equal.
So I'm using SPSS and process macro only can be used if the mediator is continuous.
So what method do you suggest? I have been searching all over the internet and no answer found, so I'd really appreciate if you helped...
Answer: One option is to assume some distribution within each category and then assign the people in that category a specific number of minutes, then use that as a continuous measure. I would try several assumptions, especially for the lowest and highest category. Use any other knowledge you have.
In a general population, I'm guessing that a lot of the people in category 1 actually spend no time at all on Instagram.
Another is to either not use that macro or not use SPSS. R and SAS both have a method called optimal scaling, which is useful for ordinal variables in regression. They have other scoring options, too. Mediation analysis can be done in a few ways but, AFAIK, all of them rely on regression and no other methods.


Using V and v chord when melody alternates between harmonic and natural minor

Question: When using the harmonic (or melodic) minor scale for a melody, it is often combined with the natural minor, depending on whether the seventh is a leading tone going up to the tonic (in which case you'd use a raised seventh), or e.g. a melody going downward (in which case you'd use the natural seventh). This sounds very "normal", and a non-musician would hardly notice that you're mixing two different scales.
However, when writing a song with a melody that alternates between harmonic and natural minor, does this alternating between a raised and natural seventh extend to the choice of V or v chord? E.g. if the overall key is E minor, would you use both B and Bm chords depending on the local context? And would that local context be the melody, or the chord progression (i.e. would you use B when the next chord is C or another chord that contains the note E, and Bm in other situations)?
Comment: Whichever fits  better. Rules don't apply.
Comment: @Tim Well, I sort of made up my mind already, based on what sounds good to me, but I was wondering if there's any relevant theory or standard way of doing this.
Comment: I'm struggling to decide which answer to accept, as each contains interesting information. Give me a couple of days...
Answer: Melodic minor for up and natural for down simply is not how minor key music works. That is a short hand textbook rule.

...E minor, would you use both B and Bm chords depending on the local context?

Yes.
A lot depends on style. In classical music it really depends on whether the seventh degree is raised to act as a leading tone.
In the following lesson you can see how the raised ^7 moves to the tonic. But in bar 10 the ^7 is the lowered form, F natural. It is part of a descending line - not acting as a leading tone. Basically the raised form is used for cadential (tonic/dominant) harmony.

But that example has the ^7 degree in the bass. It doesn't quite address your question about cases when it is in the melody (treble part.)
The following is from a partimento lesson...

...where the lowered ^7 is in the melody and beat 4 of bar 1 is clearly the minor v.
The bass part is the actual content of the lesson and it opens like this...

...where the use of the raised ^7 - the G# in the bass - for the cadence is clear. At such cadence points we would definitely expect the major dominant V.
Both forms of the dominant are used in minor and the ^7 degree can be either lowered or raised in either a harmonic or melodic context and in either ascending or descending contexts. The difference depends mostly on whether the harmony is cadential, a dominant to tonic progression.
Comment: I think 'classical' harmony is where you have a very circumscribed style. To the extent you asked about "...standard way of doing this" I think what I posted about classical style is that standard. Beyond that - non-functional, parallel harmony, etc. - I think you can enjoy a lot of freedom.
Comment: I'm using a lot of step-wise changes and parallel passing chords, like v-bv-iv and V-VI-VII, instead of cadential harmony, so that's why it felt more natural to extend the rising/falling rule to the choice of chords too.
Comment: Do you mean `V bVI bVII`, so in `E` minor: `Bm C D`?
Comment: Yes, indeed; the terminology is confusing me. Bm-Bbm-Am and B-C-D in the key of Em. The choice of B is determined by the melody at that point, but I was wondering whether you can use a "v when falling / V when rising" rule for chords similar to "^7 when rising / 7 when falling" for melody.
Answer: When you write a melody, you imply something about the harmony as well, and so you have some kind of a vague idea or gut instinct about chords, even if you can't or don't want to explicate any specific chords right away. You can do it either way. Think of chords first (for example by using voice-leading as a guideline for thinking up something) and place melody notes on strategic places in the harmony progression, fulfilling any other criteria like ups and downs, tension and release, rhythmic feel, expectation and surprise, etc. Or write melody notes straight away for example following the ups and downs and natural rhythm of lyrics, and then think of nice chords to support it while satisfying other "design criteria" like tension and release etc. Modern pop songs usually stick to colorless/tasteless/odorless minor or major pentatonic scales for melody, and backing chords live their own separate life.
In any case, if you want to do a Bm feel, you'll use a D note in the melody. Or if you use a D# note in the melody, you won't harmonize it with a Bm chord. Use both if you like.
Here's one in Em, with both Bm and B, chords first, placing melody notes on critical spots, with a rhythm. I guess it could go melody first as well, but if you write this melody without chords, you won't need a musical genius to come up with these chords.

Edit: I'll add my comment about the "going up" vs "going down" thing here. What happens with the melodic minor scale when going up is, it implies chords, namely secondary dominant - dominant - tonic. For example if the key has been established as Em, then even if played without accompaniment, the melody notes B - C# - D# - E feel like chords B - F#7 - B7 - Em. For some reason the melodic minor scale is often explained as some kind of natural phenomenon or semi-physical thing in itself that "just works" when going up, but IMO it's like a common short melodic phrase, with chords virtually embedded in it. It outlines a chord progression.
Why the melodic minor is not used all that often in descending melody phrases is that it requires a bit more imagination to feel a sensible harmony progression there. For example Em - B7 - F#7 - Em makes much less sense than Em - F#7 - B7 - Em. What comes to the harmonic minor scale's B - C - D# - E, it sounds like e.g. the chords Em - Am - B7 - Em. And descending E - D# - C - B works too: Em - B7 - Am - Em is easy to imagine there.
Comment: @YourUncleBob IMO what happens with the melodic minor scale when going up is, it implies chords, namely secondary dominant - dominant - tonic. For example if the key has been established as Em, then even if played without accompaniment, the melody notes B - C# - D# - E feel like chords B - F#7 - B7 - Em. For some reason the melodic minor scale is often explained as some kind of natural phenomenon or semi-physical thing in itself that "just works" when going up, but IMO it's like a common short melodic phrase, with chords virtually embedded in it. It outlines a chord progression. :)
Comment: In parts of the song where the melody isn't using a D or D#, and there isn't a clear melodic reason to choose a Bm or B chord, would the rising/falling direction of the chord sequence determine the choice of V or v, i.e. Am-B-C vs. C-Bm-Am?
Comment: @YourUncleBob The root note is only one of the voices, and I don't think the rising/falling thing has much to do with it anyway. In simplified terms, movement means change in position, and I feel the "position" of harmony in terms of (1) where might the tonic be, and (2) are we home or leaning to the subdominant or dominant side. The choice between B or Bm should be about how strong dominant feelings you want. Even the D# note alone, or any of A/C/D#/F# dim7 chords, or F13 etc. can work as a dominant, it doesn't need the B note.
Answer: In lots of classical pieces the composer follows a not-so-obvious (to me) rule (it does make sense.) Ascending scale like use the raised forms of steps 7 and 8 and descending forms use the lower forms; at the same time, the minor v is used in non-cadential situations; the major V in cadential ones. Of course,  the classical composers used the lower step 6 and raised step 7 commonly.
A couple of examples can be observed here. When a cycle of fifths (or fourths or circle depending on one's naming preference) is repeated, the v minor may be used in the first cycle and the V major at the end: i-iv-VII-III-VI-ii0-v-i-iv-VII-III-VI-ii0-V-i (or I). Similarly there are some renditions (can I say this for 1600s music?) of Greensleeves that go i-VII-i-v-III-VII-i-V-i.
About the only permutations that I have not seen (at least until the middle Nineteenth-Century) are the augmented chord on 3  and the use of a raised 6 as a neighbor tone to step 5.
What one sees (that may seem against the rules) are false relations between the raised 6 and 7 against the lowered versions in a bass vs melody.
Of course, the classical composers tended to think of the with the minor mode as a single object two mutable tones. They didn't separate the natural, harmonic, and melodic minors as objects in themselves; that's a more modern interpretation.


Is there any library for least absolute deviation (LAD) regression with regularization terms?

Question: We know that LASSO and ridge and ElasticNet all apply regularization terms on the coefficients of least squares regression. However, I have not yet found any R / python libraries that compute regularization of of Least Absolute Deviation (LAD):
$$\sum |y-X\beta|+\lambda||\beta||$$
using either the $l1$ norm (equivalent of LASSO) or $l2$ norm (equivalent of ridge). Does it make sense to apply regularization to LAD?
Comment: Please register &/or merge your accounts (you can find information on how to do
this in the **My Account** section of our [help]), then you will be able to
edit & comment on your own question.
Comment: Note that with LAD regression and an L1 penalty, you can use the same trick to turn it into a pure LAD regression problem as you can use with least squares regression and an L2 penalty: augment the data with several observations with target variable value 0 and right hand side variable values equal to $\lambda$.  Not efficient, but at least it works.
Answer: Check out rqPen and hqreg packages in R which claim to perform quantile regression with lasso and elastic net respectively. Maybe you know this already but least absolute deviation regression is median regression or quantile regression at the 50% percentile. Minimizing the absolute deviation results in the median (with potential problem of multiple solutions), same as minimizing the squared deviation results in the mean.
Answer: If you use the SGDRegressor in scikit-learn with the epsilon_insensitive loss function specified and the epsilon value set to zero, you will get a model equivalent to LAD with L2 regularization.
Answer: QuantileRegressor in scikit-learn is (rather heavily, by default) regularized.


Is the function $e^{-Bx^2}$ a contraction on $\mathbb{R}$? ($B>0$)

Question: The tittle says it all. I think it's true, and I tried to prove it by showing that the derivative of this function: $-2Bxe^{-Bx^2}$ is bounded from above with a bound less than 1, in order to do that, I tried to use Taylor series of $e^{-Bx^2}$, but it seems that leads nowhere. Any suggestion?
Here $B>0$ is a real number and we consider the euclidean norm.
Comment: So is B and x real numbers or matrices?
Comment: Also depends what norm you use.
Comment: Thanks. I'll add more details.
Comment: Hint: $x^2$ is always positive and exponential is monotonely increasing everywhere.
Answer: You want to check whether the maximum of $|f'|$ is less than $1$. So take one more derivative:
$$f''(x)=-2Be^{-Bx^2}+4B^2x^2e^{-Bx^2}.$$
This is zero if and only if $4B^2x^2-2B=0$, i.e. $x^2=1/(2B)$. At these points you have $|f'(x)|=2^{1/2} B^{1/2} e^{-1/2}$. You can check that these must be the points where $|f'|$ is largest since $f'(0)=0$ and $f'(x) \to 0$ as $x \to \pm \infty$. It is clear that this grows without bound as a function of $B$, so $e^{-Bx^2}$ cannot be a contraction for all $B$.
Comment: You are wrong on the second differentiation and also why do you even look on the second derivative?
Comment: @mathreadler I am attempting to maximize $|f'|$. Also I had a typo in the first line that I had corrected in the rest of the exposition.
Comment: I think he means contraction as in |f(x)| < |x|, i.e. operator norm $\leq 1$
Comment: Mathreadler, I don't mean that when I say contraction. I talked about that in a previous comment.
Comment: @Ian I understand your point. The problem is that I need this result for ALL $B>0$. I must say that maybe what I've claimed is not true (that's why I've said "Is the function... a contraction?")
Comment: @Devilathor I didn't finish the problem. The maximum of $|f'|$ is $2^{3/2} B^{3/2} e^{-2B^2}$. You now need to maximize this and show that it is less than $1$. This problem is of a similar character to the previous optimization problem.
Comment: I'm trying to solve an exercise about roots of the function $x+e^{-Bx^2}cos(x)$ so I translated that into the language of fixed point. In order to use the Bannach Fixed Point Theorem, It suffices to show that $e^{-Bx^2}$ is a contraction
Comment: @Ian. Wouldn't this process lead to an "infinite" chain of optimizations?
Comment: @Devilathor No, once you've optimized over both variables you have just a number, not a function. Wolfram Alpha tells me this number happens to be $\left ( \frac{3}{2e} \right )^{3/4} \approx 0.64$, and is attained at $B=\frac{\sqrt{\frac{3}{2}}}{2} \approx 0.61$.
Comment: @Ian. Of course. I don't know what was I thinking. I'm trying your suggestion but I couldn't maximize that function by hand.
Comment: @Ian. So, according that data, the function is a contraction indeed
Comment: @Devilathor Correct. Note that your problem is harder than you might think. $x+e^{-Bx^2} \cos(x)=0$ if and only if $-e^{-Bx^2} \cos(x)=x$, so you actually need to show that $-e^{-Bx^2} \cos(x)$ is a contraction. And this function is actually *not* always a contraction, so your problem is somewhat more difficult. (Cf. http://www.wolframalpha.com/input/?i=maximum+of+d%2Fdx+e^%28-x^2%29+cos%28x%29 ) You can still use the Banach Fixed Point Theorem in this context, but you need to choose an appropriate compact subset of $\mathbb{R}$ where your function really is a contraction ($[0,1]$ is fine).
Comment: @Ian I noted that, but $-e^{-Bx^2}\mbox{cos}(x)$ will be a contraction if $-e^{-Bx^2}$ is, isn't that right?
Comment: @Devilathor Not globally. The first product rule term, where the derivative hits the exponential, will be less than $1$ in magnitude. But the other term, where the derivative hits the cosine, may be big enough to push the whole thing larger than $1$ in magnitude. My Wolfram Alpha link above shows that this actually does happen when $B=1$ at $x \approx -0.6$.
Comment: I thought in restricting the domain too, but the exercise asks to prove that there is ONLY ONE root over all reals. I think that all this discussion shows that using Bannach Fixed Point Theorem is not the way to attack this problem. However, that's what occurred to me and I can't see any other way to solve it.
Comment: @Devilathor Ultimately it boils down to the Banach fixed point theorem, but you need to be sure that there aren't any roots *outside* your compact set. But this isn't too bad. There can't be any roots outside $[-1,1]$, because $|e^{-Bx^2} \cos(x)| \leq 1$. There can't be any roots in $[-1,0]$ either, because $e^{-Bx^2} \cos(x)>0$ on $[-1,1]$. So if there is a root, it is in $[0,1]$. Now you need to shrink the domain a little bit more (because the derivative is too big near x=0.59) to get it to be a contraction.
Comment: @Devilathor You also, unfortunately, must show that it maps the restricted domain into itself, which will probably not be a trivial matter.
Answer: Suppose $f$ is a contraction with rank $\lambda <1$. In particular, we must
have $|f'(x)| \le \lambda$ for all $x$, or
equivalently
${2B|x| \over e^{B x^2}}  \le \lambda$ for all $x$.
If we pick $B> ({e \over 2})^2$, then
$f'(-{1 \over \sqrt{B}}) = { 2 \sqrt{B} \over e} > 1$.
More specifically, let $B=e^2$, then $f'(-{1 \over e}) = 2$.
Comment: So your point is that the derivative cannot be bounded by 1 for all $B>0$?
Comment: @Devilathor: Yes, for sufficiently large $B$, $f$ is not a contraction. To be more specific, if you choose $B=e^2$, then $f'(-e^{-1}) = 2$.
Comment: Did you read the discussion of Ian's answer? There, it seems that what you say is not true...
Comment: Well, just do the computation in my previous comment.
Comment: ...Hm, one of us has made an error, and I am not sure which...
Comment: I didn't want to offend. I did the computations and indeed, what you say is true. So yes. there's a mistake around...
Comment: No, I think I did, I'm just not sure what it is. At any rate, Devilathor has actually reduced his original problem to a problem which is not equivalent to his original problem. I would propose that Devilathor open another question to deal with his original problem about finding roots of $x+e^{-Bx^2} \cos(x)$, i.e. fixed points of $-e^{-Bx^2} \cos(x)$.
Comment: Indeed it seems I did make a mistake: http://www.wolframalpha.com/input/?i=maximum+of+2xye^%28-yx^2%29%2Cy%3E0 My technique works, I think I just made an algebraic mistake. I've edited my answer to remove the part where it says it is a contraction. My comments on my answer will remain problematic...
Comment: @Devilathor: I think my computation is correct.
Comment: I found my mistake: I turned $4B^2x^2-2B=0$ into $x^2=2B$ instead of $x^2=1/(2B)$.
Comment: I posted my original problem as a question by itself: http://math.stackexchange.com/questions/1182748/show-that-xe-bx2-mboxcosx-has-only-one-root-over-all-reals-b0 I hope some of you can help me. Ian, your answer was very important, of course, you get the accepted one! Thank you very much.
Comment: @Ian: 2B or not 2B?
Comment: @copper.hat Nice one!!
Comment: Hmmm, strange the way this works...