HTTPS / HTTP mod_rewrite rules, strange interaction with subsequent CMS index.php rewrite

Question: With the below rewrites in htaccess, as expected, this HTTPS request is not rewritten:
https://example.com/system/anything
is not re-written to
http://example.com/system/anything
But, unexpectedly, this HTTPS request is rewritten:
https://example.com/preview/anything
is re-written to
http://example.com/index.php/preview/anything
Why is this?
A couple other facts / observations:
/system/ is an actual path on the server. But /preview/ is not an actual path--it's a URL segment that makes sense in the CMS, e.g., /index.php/preview/anything is how the CMS gets the request for the /preview/anything URL.
Other non-system URLs do rewrite properly (from HTTPS to HTTP), and properly do get passed to index.php. E.g.,
https://example.com/real
is re-written to
http://example.com/real
Here is the full block of rules:
 <IfModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /

 # Force HTTPS for System URLs
 RewriteCond %{REQUEST_URI} ^/system(.*)$ [NC]
 RewriteCond %{HTTPS} !=on
 RewriteRule ^(.*)$ "https://example.com/$1" [R=301,L]

 # Force HTTP for Other URLs, but not: system or preview
 RewriteCond %{REQUEST_URI} !^/(system|preview)/(.*)$ [NC]
 RewriteCond %{HTTPS} =on
 RewriteRule ^(.*)$ "http://example.com/$1" [R=302,L]

 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteRule ^(.*)$ /index.php/$1 [L]
 </IfModule>

Any insights into why /preview/ gets the weird treatment?

Added: Note that /preview/anything is getting a 302 redirect to /index.php/preview/anything -- and THAT'S a big part of what seems so weird / unexpected. It should not be getting a redirect in the final rule, just a rewrite.
Comment: Just found a similar, also unanswered, question: http://serverfault.com/questions/128554/order-of-mod-rewrite-rules-in-htaccess-not-being-followed
Answer: Are these rewrite rules in an .htaccess file? In this case the [L] flag does not do what you think it does — it stops processing of the current ruleset, but then the request is processed by Apache again, using .htaccess files appropriate for the rewritten URI, therefore your rules may be executed again. This does not happen for rules which are in the Apache configuration file (and not inside a <Directory> section) — in this case the [L] flag is handled as expected.
For your example, https://example.com/preview/anything is internally rewritten into https://example.com/index.php/preview/anything by the third rule; however, in order to process that request, Apache must read the .htaccess file again — and this time the URI matches your second rule, which returns a 302 redirect.
Apache 2.4.x supports the [END] flag which stops such rewrite loops, unlike [L]; solutions for earlier Apache versions are more complex.
If you want to make sure that Apache makes only a single pass over your rewrite rules, you may add the following rule before all others:
RewriteCond %{ENV:REDIRECT_STATUS} !=""
RewriteRule ^ - [L]

On the first pass REDIRECT_STATUS will be empty; on the second pass it will have a non-empty value (usually 200), and the rule will match and really stop further rewrites.
In case such a rule is not appropriate (e.g., in some cases you need to handle rewritten URIs on the second pass), you can set an environment variable in rules which should be really final:
RewriteRule ^(.*)$ /index.php/$1 [L,E=FINISH:1]

and add the following rule before all others:
RewriteCond %{ENV:REDIRECT_FINISH} !=""
RewriteRule ^ - [L]

Note that during the second pass Apache prepends REDIRECT_ to environment variable names which were defined during the first pass, therefore you need to set FINISH, but test REDIRECT_FINISH.
Alternatively, you may try to modify your matching conditions so that on the second pass URIs modified on the first pass will not be matched (e.g., insert (index\.php/)? into the regexp in the second rule).
Comment: Yes, the `L|Last` flag means *stop processing rules*, but as soon as the result of the rule is an alias (and the third rule makes an alias) this alias becomes a new entry for the rewrite engine, and all rules are applied on this new entry.
Comment: Yes! I totally see it now--and, it actually seems obvious, in retrospect. While I knew that the results of the 301/302 redirects would be run through the rewrite engine again, I didn't fully grasp that the result of the alias (third rule) would be as well. I had already fixed this in production by blocking `index\.php/preview` in the second rule, and it's great to now know about the options of using `REDIRECT_STATUS` or combo of `E=FINISH` and `REDIRECT_FINISH`. Thanks!
Answer: This sounds like the last rule is introducing an absolute http-based redirect when it's being done, perhaps due to some bug or feature in Apache or mod_rewrite.
How about you break it down into two rules with an extra condition?
Try this:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTPS} =off
RewriteRule ^(.*)$ http://example.com/index.php/$1 [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTPS} =on
RewriteRule ^(.*)$ https://example.com/index.php/$1 [L]

Answer: Is the http scheme (instead of https) that bothers you? If so, you should look at the RewriteBase directive.
From the mod_rewrite docs:

The RewriteBase directive specifies the URL prefix to be used for per-directory (htaccess) RewriteRule directives that substitute a relative path.
This directive is required when you use a relative path in a substitution in per-directory (htaccess) context (...)
Comment: Thanks, but no. RewriteBase just has to do with the path, right? It doesn't recognize the protocol / scheme.
Answer: I don't think you need the '(.*)$' at the end of the RewriteCond statements, because you're not using the captured data anywhere.  You could simplify those 2 like so:
RewriteCond %{REQUEST_URI} ^/system [NC]

and
    RewriteCond %{REQUEST_URI} !^/(system|preview)/ [NC]
What I would also recommend is turning on RewriteLog and setting RewriteLogLevel to see exactly what apache is doing per request.  With apache 2.2 or less, it would be:
RewriteLog /var/log/apache2/rewrite.log
RewriteLogLevel 7

You should see step by step exactly what matches and what action apache took.


Upgrading Active Directory

Question: My company operates a SaaS product. My charge is our corporate environment, i.e. users, computers and the Data Center at the Corp office. I am planning to upgrade our current Active Directory structure. I have roughly 65 users where 60 work from our Corp office and the other five from their respective Home Offices. I have five data centers geographically dispersed with their own AD Forest. I would like to pull everything under a single forest. What is best practice here? Should they be their own independent Forest or should they be Domains under a parent forest. Keep in mind all access to the other data centers are accessed from the Corp office.
Also would it be better for me to Upgrade my Current Active Directory environment by promoting up or start from scratch? What are the pros and Cons of both?
Answer: Typically once you get to AD, you want to promote up, not migrate. That way you don't have to do the computer account migrations, etc. The question of whether or not to be separate forests (or even domains) is more dependent on how you function, than anything else. Some things to think about:

Domains are not a security boundary. Forests are. If you require security boundaries that cannot be handled via delegated rights, then you should use multiple forests. Otherwise, one forest will do.
Do you need multiple domains? Or can you better served by using OUs and delegating rights at the OU level?
If you're worried about authentication traffic over the WAN and things like that, you should already have physical sites set up within AD. This will ensure that folks initially use the closest domain controller.
With 5 users geographically dispersed across multiple sites, do you need to deploy a DC to them (how's network connectivity, outages, etc.)? If not, then that gives more reason to pull back to a single forest, single domain configuration. Even if they do have issues with network connectivity, you can still deploy a single DC to the site. If you're going to 2008, this can even be a read only DC.
Answer: The crux of your question sounds like: Do I want a multi-forest infrastructure or a single forest infrastructure?
You want multi-forest when you need security boundaries between the organizations. Domains are not security boundaries.
Answer that question, and then the rest should be easy. You want as few forests as possible to satisfy your security boundary requirements. You want as few domains as possible to satisfy your replication boundary requirements.
Answer: My understanding is a bit dated, but newer versions of AD handle this case a bit better. I think this is a case where an empty root is a good idea. You have a domain at the top of your forest (say... dir.organisation.org) that is only there for DNS and global catalog. You then create sub-domains (tx.dir.organisation.org, ca.dir.organisation.org, etc) as child domains of the empty root. Then configure Sites for each of the geographically dispersed data-centers. Global catalog data will still have to be replicated across your WAN links, and so will DNS updates to some extent. Whether or not this is reasonable for your environment, you'll have to decide for yourself. But if you have the infrastructure for this, it is what I recommend. This will provide the built in trusts and unified account database that are one of AD's primary advantages.
As for upgrading, we're a fan of promoting the functional level of the domain, then using dcpromo to demote a DC to a member server, reformatting it with the new OS version, then using dcpromo to make it a DC again. Wash, rinse, repeat for each DC until all are upgraded. Then once you're comfortable that your compatibility problems are solved, raise the functional level to that of the new OS.
Comment: Empty root doesn't effectively do anything any longer. Where that was once the recommended practice, once it was shown than being a domain admin of any domain in the forest means you can escalate rights to take over the forest, that has stopped being the recommended practice.

Upgrading Active Directory


Polynomials with specified ranges in intervals

Question: Say I have two finite intervals $[a,b],[c,d]\subsetneq\Bbb R$ where $a<b<c-1<c<d$ and $b-a=d-c=s<1$.
I want to find a polynomial $f \in \Bbb R[x]$ such that $$\forall x\in[a,b],\mbox{ }f(x) \in\big[\frac{a+b}{2} - r, \frac{a+b}{2} + r\big]$$ and $$\forall x\in[c,d],\mbox{ }f(x) \in\big[\frac{c+d}{2} - r, \frac{c+d}{2} + r\big]$$ for some fixed $r\in(0,\frac{s}{2})$.
Then what is the least degree polynomial I can use?
Do Chebyshev polynomials help here? I know that for each interval I can find two scaled and shifted Tchebyshev polynomials to get what I want separately in each interval. However to find one polynomial that works on both intervals is difficult. I can try to do a weighted sum of the separate Tchebyshev polynomials. However then I am unable to guarantee what I want.
Let $T_n(x)$ be Tchebyshev polynomial of some degree $n$ which takes value between $(-1,+1)$ for $x\in(-1,1)$.
Then $$f_{a,b}(x)=\frac{a+b}{2}+r\cdot T_n\big(-\frac{2}{a-b}x+\frac{a+b}{a-b}\big)$$ and $$f_{c,d}(x)=\frac{c+d}{2}+r\cdot T_n\big(-\frac{2}{c-d}x+\frac{c+d}{c-d}\big)$$ does the job for each interval separately.
Can I build $f(x)$ from $f_{a,b}(x)$ and $f_{c,d}(x)$ by using some weighted sum. What would be the degree of the resulting polynomial?
Answer: Hint:
A degree five version might work. Symmetrize things around zero, and rescale so the two midpoints are at $1,-1.$ Insist on $f(x)=ax+bx^3+cx^5,$ which since it is odd means one only needs to look at the interval containing the midpoint $1$. So if $s<1<t$ are the two ends of the interval $[s,t]$ with midpoint $1$, we will send $1$ to itself, and then choose some interval $[m,n]$ around $1$ for the image, and we may as well have the midpoint of $[m,n]$ be $1$ also. We also choose $m,n$ close enough to $1$ that the interval $[s,t]$ gets shrunken by the appropriate amount to satisfy the conditions.
Now one can check that the three equations $f(1)=1,\ f(s)=m,\ f(t)=n$ lead always to uniquely solvable equations for the coefficients $a,b,c.$ [This is a matter of looking at the determinant of the system.] If one is lucky the resulting curve will end up increasing on the interval $[s,t],$ and that interval will map into the interval $[m,n].$
As simple-minded as it seems, this scheme did indeed work when I took $s=0.8,\ t=1.2$ and went for a shrink by a factor of $1/2$ via $m=0.9,n=1.1.$ When I did this, the coefficient of $x^3$ came out negative, however the function when graphed was increasing. Likely the derivative, a quadratic in $x^2,$ had negative discriminant, though I didn't check that.
This may not always work, and to even look at whether it does would involve perhaps prohibitive symbolic algebra, in order to get sufficient information on the coefficients $a,b,c$ in terms of the chosen parameters $s,t,m,n.$


Google Cloud Function - SendGrid - 400 Bad Request

Question: I'm trying to send an email using SendGrid in my Google Cloud Function. I have my key in the Firebase environment variables, so that's present.
const sgMail = require('@sendgrid/mail');
sgMail.setApiKey(SENDGRID_API_KEY);

Here is my GCF .ts code:
Code
const msg = {
    to: to,
    from: from,
    subject: 'Email Subject',
    content: [{
        type: "text/html",
        value: body
    }]
};

await sgMail.send(msg)
    .then(r => {
        console.log(r);
    });

When I trigger the function and check my logs, this is the error I get:
Error
error:  { Error: Bad Request
    at ResponseError (node_modules/@sendgrid/mail/node_modules/@sendgrid/helpers/classes/response-error.js:19:5)
    at Request.http [as _callback] (node_modules/@sendgrid/mail/node_modules/@sendgrid/client/src/classes/client.js:124:25)
    at Request.self.callback (node_modules/@sendgrid/mail/node_modules/request/request.js:185:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (node_modules/@sendgrid/mail/node_modules/request/request.js:1161:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (node_modules/@sendgrid/mail/node_modules/request/request.js:1083:12)
    at IncomingMessage.g (events.js:292:16)
code: 400,
message: 'Bad Request',
response:
{ headers:
    { server: 'nginx',
        date: 'Sun, 16 Dec 2018 16:27:56 GMT',
        'content-type': 'application/json',
        'content-length': '402',
        connection: 'close',
        'access-control-allow-origin': 'https://sendgrid.api-docs.io',
        'access-control-allow-methods': 'POST',
        'access-control-allow-headers': 'Authorization, Content-Type, On-behalf-of, x-sg-elas-acl',
        'access-control-max-age': '600',
        'x-no-cors-reason': 'https://sendgrid.com/docs/Classroom/Basics/API/cors.html' },
    body: { errors: [Object] } } }

Does anyone have any familiarity with this error? It mentions CORS, but that makes no sense, because it's a cloud function, not browser. The SendGrid API isn't that great, it mostly goes over the field names and provides no examples. Thanks for any help provided!
EDIT
Just to update the question, in the front end response I send myself I found a different error than the console.log(error) in the GCF logs:
"Email failed: Bad Request (400)
The from object must be provided for every email send.
It is an object that requires the email parameter,
but may also contain a name
parameter.  e.g. {"email" : "firstname.lastname@example.com"}  or
{"email" : "email@example.com", "name" : "Example Recipient"}.
    from.email
    http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#message.from"

EDIT 2: SOLUTION
My from email source comes from a document in Firestore, and I was trying to get a field that didn't exist because I was querying the wrong document, therefore from was undefined. Corrected it, as well as removed "content" in the msg object per the suggestion / answer below, and everything works fine.
Answer: According to the documentation ...the msg has the wrong elements:
const sg = require('@sendgrid/mail');
sg.setApiKey(process.env.SENDGRID_API_KEY);

const msg = {
    to: 'email@example.com',
    from: 'email@example.com',
    subject: 'Sending with SendGrid is Fun',
    text: 'and easy to do anywhere, even with Node.js',
    html: 'and easy to do anywhere, even with Node.js',
};

sg.send(msg).then(() => {
    /* assume success */
})
.catch(error => {

    /* log friendly error */
    console.error(error.toString());

    /* extract error message */
    const {message, code, response} = error;

    /* extract response message */
    const {headers, body} = response;
});

Comment: Thanks for the answer, nice find on that documentation. I changed to that format. I did, however, find a new error that I put at the bottom of an updated question. So, I'm investigating. I'll keep this Q/A updated.
Comment: Hahah, I had a malformed db call to the wrong document in Firestore, therefore my "from" was undefined. But, I also probably needed the format for `text` and `html` you posted above. Thanks for the nudge!
Comment: @Kenny not certain about it, but one possibly has to follow a format alike `email: {"email": "firstname.lastname@example.com"}` for the property names. the whole error codes somehow all use the same example. https://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html there's also a troubleshooting guide: https://github.com/sendgrid/sendgrid-nodejs/blob/master/TROUBLESHOOTING.md (the source code might also show how the `msg` object & properties need to look alike)


I need to read the epub file in ionic application

Question: I tried to read the epub file using epub.min.js but I didn't get an output.
In body tag I wrote the code:

‹

›

Answer: Are you getting any console errors? Try playing around with this file path:
var Book = ePub('../epub/ANewDay');

if your epub/ directory is a direct child of www/ try "epub/ANewDay" for your file path. Also check file extension.
Comment: i didn't got any error in chrome developer tool.
Is there any plugin available for read an epub file.
Comment: I got an below error sometine.
Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource


Session to connect Snowflake from Lambda

Question: I'm developing a AWS Lambda in python which will trigger by API Gateway and lambda will connect my snowflake. I'll process few CSV files via API Gateway to get some data from snowflake. Currently I'm using Python connector to connect Snowflake.
My issue is, if my csv has 100 records so it process the records recursively and it connects snowflake from lambda every time to process each record and its impacting on the performance.
Is there any method or mechanism that lambda can create a session for certain period of time and process all records in single connection.
Answer: As far as I know, connect() will automatically create a session that will last for a period of time. Once connected, you can use the cursor to execute multiple commands without needing to call connect() every time. Docs here. But I'm guessing you know this, and what you want is a single command instead of having to call multiple INSERT.
This is also possible, using a STAGE and COPY INTO command instead of INSERT. You can find an example from Snowflake documentation for bulk loading from AWS S3 here.
Comment: Hi @Register Sole - Thanks for the input. Its working fine. Can you please let me know how long session will be live by default and how we can modify the time if its possible.
Comment: @SuryaPratap Referring to [this](https://docs.snowflake.com/en/user-guide/session-policies.html#:~:text=If%20a%20session%20policy%20is,must%20authenticate%20to%20Snowflake%20again.), the default idle timeout is 4 hours. You can manage session policy, for example, using [this command](https://docs.snowflake.com/en/sql-reference/sql/create-session-policy.html).


Select multiple columns by name in R

Question: I'm relatively new to R, so maybe this is really basic. But I need to select multiple columns from a data frame by name in R. I know the columns I want are all adjacent. I know the name of the first column and the last column. And in my case in particular all columns have names of the form XYYYY where 'X' is the letter X and the 'Y's are numbers representing years, but some years may be omitted (I don't know whether or not any of the years in my range are omitted just that the data set is missing some years.) Is there a way to do this that doesn't involve looping an selecting each column individually until I get to the last one?
Comment: Welcome to SO. This is a very basic R question and SO is not really the place for "learning R from scratch". The core use of SO is to post data and code and then indicating the components you tried and did not work. This falls out of scope of that unless you can review the guidance here — https://stackoverflow.com/tags/r/info — and post a reproducible example (data + code) of something you tried.
Comment: This might fall out of the scope of what is appropriate for SO, but it is most certainly not a dupe of the question hrbrmstr posted.
Comment: Look into dplyr's select function. It can be combined with the starts_with function to select all columns that start with X, for example.
Comment: This is a dup of https://stackoverflow.com/questions/24250878/selecting-multiple-columns-in-data-frame-using-partial-column-name (someone who can still vote to close it should do that as a dup)
Comment: @JackSurine - maybe something like - https://stackoverflow.com/questions/29219939/subset-a-vector-of-column-names-by-a-particular-sample-prefix ?
Answer: Use select function, you can either use indices or column names for selection
for example,

for selecting 1 and 2 columns use: select(1:3)
for selecting columns by names: select(a,b) [ Note: a, b are column names]
Comment: You've just told someone very new to R to use a function — `select()` — that's not in base R and have not provided which package they should try to use to get that into their namespace. Furthermore, you did not answer their core question which is how to select columns with similar naming-structure without hand-specifying all of them. This is confusing and unhelpful for the OP.
Comment: Sorry that I forgot that you are using the baseR, you can use subset functionality. please refer to https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/subset
Comment: How is `subset()` going to help with multiple column selection by partial name matching?

Select multiple columns by name in R


show $\int f_kd\mu\leq C$ for $f_k\geq0$, $\int fd\mu\leq C$

Question: Let $(\Omega, \mathcal A,\mu)$ be a measure space and $f_k\rightarrow f$ a.e., $f_k\geq0$  and $\int f d\mu\leq C$ for some $C>0$.
How can you show $\int f_k d\mu\leq C$ ?
My attempt: I thought about using Fatou:
$$
C\geq\int fd\mu\geq\liminf\int f_kd\mu
$$
But this way I only know the limes inferior is $\leq C$.
Answer: If we are using Lebesgue measure on the unit interval, $[0,1]$, we can demonstrate a quick counter example.
Suppose that we have the sequence $f_k(x) = 1+1/k$ where $x\in[0,1]$. This converges a.e. (well everywhere) to the constant function $f(x)=1$ for all $x \in [0,1]$.
Then $\int f_k d\mu =1+1/k$ but $\int f d\mu = 1$. Thus for all $k$ we have $\int f_k d\mu > \int f d\mu = 1$.
Thus we see that $\int f d\mu = 1 = C > 0$, however for all $k$ we have $\int f_k d\mu > C$. Thus showing $\int f d\mu \le C$ is not sufficient for demonstrating $\int f_k d\mu < C$ for sufficiently large $k$.
Comment: How is this a counterexample? Couldn't I say that C = 1, which bounds both integrals in this case?
Comment: I think the way to think about this is that $C$ is unknown to $f_k$ beforehand. All we know is that $C$ bounds the integral of the limiting function, and then we look back at the integrals of the sequence and ask if they are under $C$ as well. At least that is how I interpret it.
Comment: The forward direction is certainly true by Fatou. However, the backwards direction does not hold generally.
Comment: The question needs to be more specific here. Are we asking if this can hold for all $k$? Or for sufficiently large $k$? As stated this is a counter example.
Comment: The idea is that for any $C > 0$ and larger than $\int f d\mu$ that the inequality holds for $\int f_k d\mu$. You would need to check more than $C=1$ but also $C=1/2$, $C=1/4$ etc.
Comment: The question asked for the existence of such a $C > 0$ not that it holds for all $C > 0$? Maybe I misunderstood the OP's question.
Comment: Do you mean that for all $k$ or sufficiently large $k$?
Comment: @Joel I guess sufficiently large $k$.
Answer: I think this is a counterexample.
Let $f_k=\chi_{[k,\infty)}\to f=0$ everywhere. and use Lebesgue measure.
Btw, Fatou lemma states $\int \liminf f_k\le \liminf \int f_k$


Download attached files in a Zip

Question: Im doing a website for photos. The photographer uploads all the images and drupal generates a thumbnail and a resized image.
What i'm trying to do is to put a link , so the final user can download all the original files in a zip files.
Can you guys help me out?
Answer: Yes, there's a module for that (of course ;) )
Download module:

This module can compress any files attached to a node and provides a download link. To use this module you have to extract the PclZip library in /sites/all/libraries/pclzip which can be downloaded here: http://www.phpconcept.net/pclzip/pclzip-downloads.
Comment: How do I use the Download Module?
Please, do you have any instruction about it? Thanks
Comment: @user13680 - the readme needs some work, in the meantime: http://drupal.org/node/1922568#comment-7126430
Answer: There's a better suited module available for that functionality now: https://www.drupal.org/project/zipcart, put everything into a cart and download a zip. About to start using zipcart for a project myself
Edit: ended up with DocBinder because zipcart got abandoned.
Answer: PHP has a ZipArchive class.
I'm writing some code at the moment to implement this in Drupal on a node save. I'll add an example later. But for now there are examples on the ZipArchive page.

Download attached files in a Zip


What is the best way to design a database to store record with a lot of values?

Question: I want to design a database for events and track a lot of statistic about the it.
Option 1
Create one table for Events and put all my statistic column in it.  Like number of male, number of female, number of unidentified gender,  temperature that day, time it started, any fights, was the police called, and etc.
The query would be a very simple select * from events
Option 2
Create two tables, one for Events and one for EventsAttributes.  In the Events table I would store important stuff like id, event title, and start/end time.
In EventsAttributes I would store all the event statistic and link them back to Events with a eventId foreign key.
The query would look like below. (attributeType == 1 would represent number of males)
select e.*,
(select ev.value from EventAttributes ev where ev.eventId = e.id and attributeType = 1) as NumberOfMale
from Events e

The query would be not be as straight forward as option 1, but I want to design it the right way and live with the messy query.
So which option is the right way to do it, and why (I'm not a database admin, but curious).
Thank you for your time.
Comment: Option 1 sounds like a typical way to deal with this. Option 2 sounds like EAV (entity attribute value) which is an evil anti-pattern that should be implemented very rarely. Think of a table as a way to hold all properties of an entity, in this case an event. I might consider some additional tables for things that might happen more than once during an event (like police calls, injuries etc).
Comment: In the Option 2, you forgot the third table, AttributeTypes. If you hardcode this lookup into your app, that would be an evil thing indeed.
Comment: @RogerWolf You are right.  I forgot to mention that.
Comment: @SeanLange For things that might happen more than once, I plan to represent them by number.  So for police call, the column would be an `int` type.  Thanks for the input.
Comment: So you would only record the number of times the police were called? You don't need any additional details about those calls? Things like what times they were called etc...
Comment: @SeanLange Yes.  I'm just gathering basic data.
Answer: I prefer using option 2 for designing database.
In that option(2), you apply the best practice of database normalization.
There are three main reasons to normalize a database:

The first is to minimize duplicate data.
The second is to minimize or avoid data modification issues
The third is to simplify queries.

For more details, read Designing a Normalized Database
You can create views (queries) based on this normalized database to support Option (1).
In this way, database will be ready for any future scaling.
Update:
You can use the the valuable operator pivot and common table expressions (CTE) to get eventAttributes1, eventAttributes2, ...
Suppose your tables are :events and event_attributes as described below:
    events
   ----------
    # event_id
    event_title
    start_date
    end_date

    event_attributes
    -------------
    #event_id
    #att_type
    att_value

    # is primary key

    -- using table expression (it's like a dynamic view)

    with query as (
    select e.event_id, e.event_title,a.att_type, a.att_value
    from events e
    join event_attributes a on e.event_id =a.event_id
    )
    select event_id , event_title,
    [1] as eventAttributes1, -- list all eventAttributes1 numbered [1],[2],...
    [2] as eventAttributes2
    [3] as eventAttributes3
    FROM query
    PIVOT(SUM(att_value) FOR att_type IN ([1],[2],[3])) as pvt

For details on pivot read: Using PIVOT
For details Using Common Table Expressions
Comment: I prefer option 2, but how would you efficiently query the two tables so the data are combined like, `id, eventName, eventAttributes1, eventAttributes2, ...`
Comment: Option 2 is an EAV which can get incredibly complicated very quickly. This is not something I would recommend for somebody who is relatively new to the database world. Yes I absolutely agree that normalization is the best way to go but I don't think this situation needs an EAV.
Comment: @Sean Lange , Any one new to the database world, sooner will be expert and database will be complex to be normalized in the future with huge data :).
Comment: Well we don't know how much data we are dealing with here. And by no means did I ever suggest they don't normalize their database. EAV is a horrible pattern in almost every situation. Very rarely should it be used and almost never a total EAV solution. A hybrid EAV portion is generally the best approach.
Comment: I agree, To EAV, or not to EAV? Choosing your data model.https://wq.io/1.0/docs/eav-vs-relational
Comment: @SeanLange My estimate would be around 110 columns if I combine the Events information and the Event Attributes.
Comment: That sounds like an awful lot of columns to me. I suspect some normalization would make this a lot cleaner.
Comment: @SeanLang. So, for 110 columns,normalization (or EAV) fit for purpose. Is it  right?
Comment: Not quite sure your question. But 110 columns sounds like it needs to be normalized. One way to accomplish that might be by a partial EAV but that would be a last resort and not something I would do unless the data really needs that. Since we still don't know what the OP is really storing it is impossible for us to know.


Duvida com uso de `pthread` em posix

Question: Estou usando a seguinte rotina com comando da biblioteca pthread.h:
//
//  Declaração
//
pthread_t       threads[NUM_THREADS];

//
//  Criacao
//
for(...)
{
    pthread_create(&threads[i], NULL,   MainTH,     (void *)    &thread_args[i]);
}

//
//  execução
//
for(...)
{
    pthread_join(threads[i], NULL);
}

Dessa forma elas não trabalham de forma independentes, certo?
aguardam para a terminação da outra Thread
queria saber, como faco para trabalhar com elas de forma separada, independentes?
Answer: O código me parece correto.
A função pthread_create inicia a execução da thread sem bloqueio, ou seja, a thread passa a executar sem interromper o fluxo principal.
Por fim, a função pthread_join aguarda a finalização de todas as threads. Essa bloqueia o fluxo principal para que cada thread seja finalizada uma a uma.
Comment: sim, mas somente retirar esse função vai fazer que elas trabalhem de forma independentes, não ocorrerá  do programa fechar já que o main não aguardará as threads e entra chegar ao fim?

Duvida com uso de `pthread` em posix


NoClassDefFoundError on ComputeClassWriter using Eclipselink

Question: I am using eclipselink 2.4.2 and am deploying an EAR on Weblogic 10.3.5.  I am getting the following error on every Entity when I try to deploy:
The exception was: java.lang.NoClassDefFoundError: org/eclipse/persistence/internal/jpa/weaving/ComputeClassWriter
I can clearly see that the ComputeClassWriter class is in the eclipselink 2.4.2 jar and that jar is in the appropriate lib directory of the EAR.  Has anyone come across this before?  Any help is appreciated.
Comment: Try using the `Link with Editor` with the exception on your console. It will link you to from where the class is actually loading.
Answer: It's happen the same to me.
[EL Warning]: weaver: 2014-02-17 12:59:36.897--ServerSession(285072714)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--Weaver encountered an exception while trying to weave class net.xxx.MyEntity  The exception was: java.lang.NoClassDefFoundError: org/eclipse/persistence/internal/jpa/weaving/ComputeClassWriter
Comment: your response should contain a solution
Answer: I also encountered this problem recently after upgrade EclipseLink from 2.3.0 to 2.4.2 on WebLogic 10.3.6 running in development mode with jrockit-jdk1.6.0_45-R28.2.7-4.1.0
Strangely this error does not happen on production mode WebLogic with same environment.
After some research and test I found that the startup script in 'commEnv.cmd' adds '-Xverify:none' to JAVA_OPTIONS when weblogic works with jrockit in development mode:
<code>:oracle
if "%PRODUCTION_MODE%" == "true" goto oracle_prod_mode
set JAVA_VM=-jrockit
set MEM_ARGS=-Xms128m -Xmx256m
set JAVA_OPTIONS=%JAVA_OPTIONS% -Xverify:none
goto continue
:oracle_prod_mode
set JAVA_VM=-jrockit
set MEM_ARGS=-Xms128m -Xmx256m
goto continue
</code>
After removing '-Xverify:none', weaving works without error.
I assume this could be a complex class loading and class verify bug.


Pra que serve o "display: ruby" em CSS?

Question: A pergunta é basicamente essa:

Pra que serve o display: ruby em CSS?
Esse atributo é novo?
Quando usá-lo?
Comment: Pra mostrar código na linguagem lá no japonês :P :D
Comment: Quem negativou, pode dar um feedback?
Answer: display:ruby é o display padrão ta tag <ruby>. Assim como o display padrão da
 é block e do  é inline.
O Chrome parece não usar display:ruby para a tag <ruby>, mas no FireFox vc já pode conferir que funciona. Apesar disso, a tag não é tão nova assim, e consta no  Working Draft da W3C desde 2014 como vc pode conferir no link oficial https://www.w3.org/TR/css-ruby-1/

A anotação Ruby é uma forma de anotação interlinear, consistindo em pequenas sequências de texto ao lado do texto base. Eles são normalmente usados em documentos do Leste Asiático para indicar pronúncia ou fornecer uma anotação curta.

Ou seja a tag ruby é usada como marcação de texto para indicar a pronuncia de algum texto em outro idioma, normalmente Asiáticos. Além disso, a tag não deve ser usada sozinha, ela deve ser usada em conjunto com as tags <rp> e <rt> para auxiliar na construção da marcação. Um elemento <rp> deve incluir cada um dos parênteses de abertura e fechamento que envolvem o elemento <rt>que contém o texto da anotação. O <rp> é opcional e ajuda a formatar o texto quando o navegador não tem suporte ao ruby.
Junto com a tag ruby vc pode usar também outras propriedade CSS como ruby-align e ruby-position para controlar como quer mostrar visualmente a anotação do texto. Aqui tem mais detalhes tecnicos https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Ruby

Veja o exemplo

<ruby>
  漢 <rp>(</rp><rt>Kan</rt><rp>)</rp>
  字 <rp>(</rp><rt>ji</rt><rp>)</rp>
</ruby>

Outro exemplo, mais complexo um pouco e com duas anotações.

<ruby>
  <ruby>東<rt>とう</rt>南<rt>なん</rt></ruby>
  <rt lang=en>Southeast</rt>
</ruby>

Fonte: https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-ruby-element
Atualmente segundo o CanIuse esse é o suporte dos navegadores https://caniuse.com/#feat=ruby
Comment: Dá para usar esse display em outros elementos que não sejam o próprio ruby?
Comment: @AndersonCarlosWoss acredito que sim, da mesma forma que colocamos display:block na , só não sei dizer como o elemento vai se comportar, até pq para ter a renderização como visto acima na resposta vc precisa também da tag

Pra que serve o "display: ruby" em CSS?


Problema na API após fazer deploy

Question: Minha api não está funcionando após o deploy para o servidor, está retornando:

404 - File or directory not found.
    The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.

Fazendo uns testes descobri o seguinte, está fazendo a chamada assim e não da certo:
http://meuServidor/api/Livro
mas se eu fizer assim da certo:
http://meuServidor/api/api/Livro
Porque da certo quando coloco api duas vezes? não consegui fazer isso no código, aplicar api duas vezes na rota.
Comment: Você publicou ela no IIS, certo? Por acaso você criou uma aplicação dentro do "Default web site"?
Comment: `StatusCode 404` significa endereço não encontrado, sua chamada não está batendo na API!
Comment: Já verificou se rota em ambiente de produção está igual a de ambiente em desenvolvimento?
Answer: O que aconteceu:
Você criou um aplicativo dentro do seu site(provavelmente Default Web Site). Com isso é criado uma alias(No seu caso api) e para acessar sua aplicação é necessário que tenha a alias na URL(isso ocorre independente de ser uma api ou um projeto web), exemplo: http://localhost/alias/api/rota

Como resolver:
Você pode publicar sua api no "site" Default ou criar um "site" para a api, assim sendo não terá mais que passar a alias, no exemplo da imagem, criei um novo site(botão direito em Sites>Adicionar site)

Possível "problema":
Ao criar sites, cada um ficará numa porta, desta forma, se por acaso tiver uma aplicação web e uma api publicadas, cada "site" terá uma porta
Comment: Faz sentido, se esta for a resposta deixo o meu +1
Comment: O estranho era que a pouco tempo atras funcionava, enfim, fiz o que você sugeriu e criei no "Sites" e deu certo.

Problema na API após fazer deploy


Excluding a level in your categorical variable to fit your hypothesis

Question: I have a categorical variable with 3 levels: Attention Deficit, Other Types of Deficit, No Deficit. My hypothesis is only interested in comparing Attention Deficit and Other Types Deficit using regression. Is it right to exclude the subjects with no deficit from my sample when I run the regression? Meaning I will have only one dummy coded variable in my regression?
Comment: What are you trying to compare between the two samples correspond to "Attention Deficit" and "Other Types Deficit" levels of your categorical variable?
Comment: @Epaminondas  binary outcome: Pass or fail. I am using logistic regression.
Comment: You do not need to exclude "No Deficit" from your sample, since including this level will allow you in addition to compare the three levels of your variable.
Comment: @Epaminondas That is only true if you exclude the constant.
Comment: @Epaminondas I understand that I can still make comparisons between Attention Deficit and Other Types Deficit if I keep No Deficit in my sample/categorical variable, regardless of which level I used as my reference group. But I would to know if it theoretically right for me exclude No Deficit cases from my sample simply because it is not of interest/part of my hypothesis, hence I would only have 2 levels in the categorical variable.
Answer: Generally, eliminating conditions from your analysis is frowned upon. Statistically, anything is possible, but from a research perspective, it might not be.
You can contrast only the two conditions you are interested in, but include the complete analysis/data in your reporting. Here's a great paper which talks about that and other related issues in psych research https://journals.sagepub.com/doi/full/10.1177/0956797611417632
Comment: @Dave " Is it right to exclude the subjects with no deficit from my sample when I run the regression? " This definitely suggests they were looking at excluding the subjects entirely from the analysis. I wouldn't term what you are saying as "dropping a condition" but "collapsing across conditions". That way they would be included in the regression but wont get their own estimates.
Comment: The OP suggested they wanted to exclude the subjects from the regression. I don't think that's kosher at all. One could say "conditions A and B were similar / not our focal interest in this work so we consider all subjects in conditions A and B as similar" - thereby collapsing across conditions A and B. But entirely dropping condition B from the analysis would be an unethical research practice.
Comment: Dropping a condition is done to allow the dropped condition to be contained in the intercept term, but those data points are still considered in the regression.
Comment: I do not follow. Could you please clarify?
Answer: If you want to compare Attention Deficit with Other Deficit then it is convenient to exclude Other Deficit from your model. That way the parameter for Attention Deficit will be directly the comparison of Attention Deficit with Other Deficit.
In the comments you mention that you will use this in logistic regression, so lets use an example with logistic regression. Say we want to explain union membership if women in the USA with marital status, race and education (grade_c, which is centered at 12 years of education($\approx$highschool)). In particular we are interested in the comparison never married versus married. The exponentiated constant is the odds of union membership for those women who are married, white, and finished highschool, that is, have the value 0 on all explanatory variables. So for married white highschool graduates we expect to find .24 union-members for every non-union-member. The other coefficients are odds ratios, which are ratios of odds. So if one is never married than the odds will increase by a (nonsignificant) factor $1.27$, or an increase of $(1.27 - 1)\times 100\%= 27\%$. So by excluding the married dummy, the coefficient for never married is directly a comparison between the married and never married.
. logit union ib2.marst i.race grade_c, or base nolog

Logistic regression                             Number of obs     =      1,876
                                                LR chi2(5)        =      35.73
                                                Prob > chi2       =     0.0000
Log likelihood = -1027.0716                     Pseudo R2         =     0.0171

-----------------------------------------------------------------------------------
            union | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
            marst |
   never married  |   1.274812   .2173981     1.42   0.155     .9126157    1.780757
widowed/divorced  |   1.145731   .1489955     1.05   0.295     .8879514    1.478347
         married  |          1  (base)
                  |
             race |
           white  |          1  (base)
           black  |   1.567148   .1929431     3.65   0.000     1.231154    1.994839
           other  |   1.760592   .7811568     1.27   0.202     .7378863    4.200763
                  |
          grade_c |   1.100223   .0237531     4.42   0.000     1.054638    1.147777
            _cons |   .2352931   .0195246   -17.44   0.000     .1999754    .2768482
-----------------------------------------------------------------------------------

@Epaminondas mentioned that you do not have to exclude any dummy. This is true if you exclude the constant, but will dramatically change the interpretation of your coefficients. Now the exponentiated coeffcients of the marital status dummies are no longer odds ratios, but odds. So to continue the example: For married white highschool graduates the odds of being a union member is 0.30, while for never married white highschool graduates the odds of being a union member is 0.27. We can compare those odds by computing their ratio, and if we do so we get the odds ratio of 1.27 we got in the first example. So both are just different representations of the same model. Both can be useful, but if you want to focus on the comparison, then the first is a more direct way of doing so.
. logit union ibn.marst i.race grade_c, or base nolog nocons

Logistic regression                             Number of obs     =      1,876
                                                Wald chi2(6)      =     447.72
Log likelihood = -1027.0716                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------
            union | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
            marst |
   never married  |   .2999546   .0513313    -7.04   0.000     .2144812    .4194901
widowed/divorced  |   .2695827    .033444   -10.57   0.000     .2113943    .3437881
         married  |   .2352931   .0195246   -17.44   0.000     .1999755    .2768482
                  |
             race |
           white  |          1  (base)
           black  |   1.567148   .1929431     3.65   0.000     1.231154    1.994839
           other  |   1.760592   .7811567     1.27   0.202     .7378862    4.200763
                  |
          grade_c |   1.100223   .0237531     4.42   0.000     1.054638    1.147777
-----------------------------------------------------------------------------------

. di .2999546 / .2352931
1.2748126

Comment: @Marteen Buis I see that you have all three levels in both of your outputs. And the number of observation is also the same, meaning you did not remove any observation. My question was if, in this context, I want to compare only married vs never married. Is it statistically right to remove those who are widowed/divorced? Correct me if I am wrong, but the explanation you gave depends on the contrast bonding that you use for your dummy variables? So for the first model you used set last to zero?
Comment: In the first model the married category was excluded, which is equivalent to setting the log odds ratio to zero and this the odds ratio to one.
Comment: So observations that were widowed/divorced are not removed from the sample but simply set as the reference group.
Comment: No, the married are exluded from the model, so they are the reference group. Both never married and divorced are compared with the married.


The Order of the elements in G

Question: Suppose that $G$ is an group with identity element $e$. Am I correct that the order of $e$ is $1$ in all cases?
Bob
Answer: Yes. The order of an element $g$ is the smallest positive integer $n$ so that $g^n = e $. In this case that's $1$.

The Order of the elements in G


Poisson process and ship insurance with binomial damage claim

Question: A insurance company offers ship owners insurance for broken down ships. For the calculation assume that the reported broken down ships follow a poission process. On average a breakdown happens on $1$ day out of $1000$ days and the damage is $\text{Binomial($2\cdot 10^7,0.5$)}$ distributed and is independent of the damage of other ships and independent of the poisson process. Calculate the expected damage in one year ($365$ days).
I am having trouble "merging" the poisson process with the binomial damage claim. I basically have two random variables $$X:=\text{number of broken down ships} \sim Poi(\frac{1}{1000}) \\ Y:= \text{damage of broken down ship} \sim Bin(2\cdot 10^7,0.5)$$
How can I put these together to find the expected damage?
Answer: If there are a random number $X$ of damaged ships, and each one has damage $Y_i$, then the total damage is
$$\mathbf E \left[ \sum_{i=1}^X Y_i \right].$$
Wald's equation allows you to calculate the expectation of a random sum of random variables, under the condition that the terms of the sum are independent and identically distributed.
$$\mathbf E \left[ \sum_{i=1}^X Y_i \right] = \mathbf E[X] \mathbf E[Y].$$
In your case this becomes
$$\mathbf E \left[ \sum_{i=1}^X Y_i \right] = \left(\frac{365}{1000}\right) \times \left( \frac12 \times 2 \cdot 10^7\right) = 3.65 \times 10^6$$
Comment: This is the first time I have seen Wald's equation and it seems very useful. Thank you very much for your help!


showing $\zeta'(1) =0$ and $\psi(0)=0$?

Question: starting from reflection formula and digamma function
I obtain
\begin{align}
\frac{\zeta'(s)}{\zeta(s)} + \frac{\zeta'(1-s)}{\zeta(1-s)} = \log(2\pi) + \frac{\pi}{2} \cot\left(\frac{\pi s}{2}\right) - \psi(1-s)
\end{align}
taking limit goes 1, reference says that
\begin{align}
\frac{\zeta'(0)}{\zeta(0)} = \log(2\pi)
\end{align}
It seems that $\zeta'(1)=0$ and $\psi(0)=0$. I wonder how to prove this

Edit
due to @Mindlack I got $\frac{\zeta'(1)}{\zeta(1)}=0$, but still got problem with $\psi(0)=0$....
From Mathematica I see that $\psi(0) = -\infty$....
The reference I follow is Nicolas M Robles master thesis on zeta function regularization page 28
Comment: No, since there is a pole $\zeta'(1)$ is undefined! However $\frac{\zeta'}{\zeta}$ has a limit in $1$ which is $0$.
Comment: $\zeta(z) = \frac{1}{z-1} f(z)$ where $f$ is analytic and $f(1)=1$ thus $f'/f$ is analytic and $\frac{\zeta'(z)}{\zeta(z)} = -\frac{1}{z-1}+\frac{f'(z)}{f(z)}=-\frac{1}{z-1}+O(1)$
Comment: You are aware that $\zeta(z)=\frac{1}{z-1}+O(1)$ as $z \rightarrow 1$?
Comment: @Mindlack, I see what you mean. zeta has a simple pole at $z=1$ so $\zeta'(1)=0$. I get it.
Comment: @Mindlack, can you show me some details? I am confused that how you obtain $\frac{\zeta'}{\zeta} =0$ at $z=1$.
Answer: A delicate calculation that requires justification
\begin{align}
\frac{\mathrm{d} \zeta}{\mathrm{d}z} &= \frac{\mathrm{d} }{\mathrm{d}z}\left( \sum_{n=1}^{\infty}n^{-z}\right) \\
&= \sum_{n=1}^{\infty}\left(- \ln(n) n^{-z} \right) \\
&= -\sum_{n=1}^{\infty}\left(\frac{\ln(n)}{n^{z}} \right) \\
&= -\sum_{n=2}^{\infty}\left(\frac{\ln(n)}{n^{z}} \right)
\end{align}
As $\ln 1 = 0$. Your result then follows. it is delicate but there is also a need to show that the sum converges uniformly and also that the sum of the derivatives converge uniformly.
Comment: Thus $\lim_{z \to 1^+}\zeta'(z) = -\infty$

showing $\zeta'(1) =0$ and $\psi(0)=0$?


"Invalid type owner for DynamicMethod" error when sorting an Interface

Question: We're using Andrew Davey's BindingListView<T> class via sourceforge to bind collections to a DataGridView and allow sorting and filtering.
This works fine for normal collections. In one case however the collection we're binding to is an Interface type and we get this error if we try to sort on it:
Invalid type owner for DynamicMethod
The error is deep in Andrew Davies' code so it's hard for us to know where to start.
        private static Comparison<T> BuildValueTypeComparison(PropertyInfo pi, ListSortDirection direction)
        {
            MethodInfo getMethod = pi.GetGetMethod();
            Debug.Assert(getMethod != null);

            DynamicMethod dm = new DynamicMethod("Get" + pi.Name, typeof(int), new Type[] { typeof(T), typeof(T) }, typeof(T), true);
            //^^^ ======== Here's the line reporting the error=========== ^^^

            ILGenerator il = dm.GetILGenerator();

            // Get the value of the first object's property.
            il.Emit(OpCodes.Ldarg_0);
            il.EmitCall(OpCodes.Call, getMethod, null);
            // Box the value type
            il.Emit(OpCodes.Box, pi.PropertyType);

            // Get the value of the second object's property.
            il.Emit(OpCodes.Ldarg_1);
            il.EmitCall(OpCodes.Call, getMethod, null);
            // Box the value type
            il.Emit(OpCodes.Box, pi.PropertyType);

            // Cast the first value to IComparable and call CompareTo,
            // passing the second value as the argument.
            il.Emit(OpCodes.Castclass, typeof(IComparable));
            il.EmitCall(OpCodes.Call, typeof(IComparable).GetMethod("CompareTo"), null);

            // If descending then multiply comparison result by -1
            // to reverse the ordering.
            if (direction == ListSortDirection.Descending)
            {
                il.Emit(OpCodes.Ldc_I4_M1);
                il.Emit(OpCodes.Mul);
            }

            // Return the result of the comparison.
            il.Emit(OpCodes.Ret);

            // Create the delegate pointing at the dynamic method.
            return (Comparison<T>)dm.CreateDelegate(typeof(Comparison<T>));
        }

Comment: I've just downloaded the library, and modified the sample application so that it is binding to an interface rather than a concrete type and it works fine; so it can't be just the fact that you are binding to an interface. Can you post up a minimal interface and implementing class that exhibits this behaviour because this is definitely implementation specific?
Comment: OK I take that back, you are using the `AggregateBindingListView` and it does throw.
Answer: UPDATE: I've finally got this working the right way, please see below.
DynamicMethod builds a method dynamically at runtime; in the library you are using the method that is created is being added to the object T; however, when T is an interface this fails because you cannot add a method to an interface.
The main problem is in the method:
private static GetPropertyDelegate BuildGetPropertyMethod(PropertyInfo pi)
as the way this was written it will only work when the collection type T is concrete.
If you change the implementation to:
private static GetPropertyDelegate BuildGetPropertyMethod(PropertyInfo pi)
{
    MethodInfo getMethod = pi.GetGetMethod();
    Debug.Assert(getMethod != null);

    DynamicMethod dm = new DynamicMethod(
        "GetProperty_" + typeof(T).Name + "_" + pi.Name, typeof(object),
        new Type[] { typeof(T) },
        pi.Module,
        true);

    ILGenerator il = dm.GetILGenerator();

    il.Emit(OpCodes.Ldarg_0);
    il.EmitCall(OpCodes.Callvirt, getMethod, null);
    if (pi.PropertyType.IsValueType)
    {
        il.Emit(OpCodes.Box, pi.PropertyType);
    }

    // Return the result of the comparison.
    il.Emit(OpCodes.Ret);

    return (GetPropertyDelegate)dm.CreateDelegate(typeof(GetPropertyDelegate));
}

it will work for both concrete types and interfaces.
You also need to update the following two methods:
private static Comparison<T> BuildValueTypeComparison(PropertyInfo pi, ListSortDirection direction)
private static Comparison<T> BuildNullableComparison(PropertyInfo pi, ListSortDirection direction)
I might be wrong, but I think that the speed gain achieved in these methods comes from the fast property read, so there is not really that much benefit from writing the whole methods using the DynamicMethod method; we can just reuse the BuildGetPropertyMethod from above. So doing this, these become:
private static Comparison<T> BuildValueTypeComparison(
    PropertyInfo pi,
    ListSortDirection direction)
{
    GetPropertyDelegate m = BuildGetPropertyMethod(pi);
    Comparison<T> d = delegate(T x, T y)
    {
        object mx = m(x);
        object my = m(y);

        IComparable c = (IComparable)mx;

        if (direction == ListSortDirection.Descending)
        {
            return -c.CompareTo(my);
        }

        return c.CompareTo(my);
    };

    return d;
}

private static Comparison<T> BuildNullableComparison(
    PropertyInfo pi,
    ListSortDirection direction)
{
    GetPropertyDelegate m = BuildGetPropertyMethod(pi);
    Comparison<T> d = delegate(T x, T y)
        {
            object mx = m(x);
            object my = m(y);

            IComparable c = (IComparable)mx;

            if (c == null)
            {
                c = (IComparable)my;

                if (c == null)
                {
                    return 0;
                }

                return direction == ListSortDirection.Descending
                    ? c.CompareTo(mx) : -c.CompareTo(mx);
            }

            return direction == ListSortDirection.Descending
                ? -c.CompareTo(my) : c.CompareTo(my);
        };

    return d;
}

Obviously do some testing on it, but I'm pretty sure that is what you want and it should be about as fast as the previous code.
Comment: Awesome. I put your code in my test project and tried it. It works like a charm.
Answer: Do not give up and go to use DataSet! You are heading the right direction! Now, let's first take a look at the function signature:
DynamicMethod(string name,
              Type returnType,
              Type[] parameterTypes,
              Type owner,
              bool skipVisibility)

Error is "Invalid type owner for DynamicMethod"

The error message is trying to tell you, Type owner is not what the function is expecting. It wants a Class type. You are probably passing an Interface type into the Type owner. It is impossible to create dynamic methods on an Interface.
1.Error Example
Maybe you are using Dependency Injection, and you may like to use Interface.
However, this code will hit Runtime Error.
var viewModelList = GetViewModels(); //returns an IList<IViewModel> <-- has i !!
var blv = new BindingListView<IViewModel>(viewModelList);

2.Working Example
Try to redesign your code to use Concrete Type.
Now, this code will NOT hit Runtime Error
var viewModelList = GetViewModels(); //returns an IList<ViewModel> <-- has no i !!
var blv = new BindingListView<ViewModel>(viewModelList);

Then, your Sorting and Filtering on DataGridView will work automagically :)
EDIT ---------------------------------------------
P.S. About trying to redesign your code:
If you are using a MVVM/MVP pattern, think about the following logic. The IList<IViewModel> should stay on the "VM+P" side. Purpose of using IViewModel is mainly because I want to be able to replace it with, say, MockingViewModel : IViewModel for unit testing the "VM+P" side of logics.
Now, the BindingListView<ViewModel> should really be on the "V" side, that is, the YourView : System.Windows.Form { ... }. And it will be binded to the binding source from there YourBindingSource.DataSource = blv; Since I won't be unit testing a WinForm, any logic in it I will try to refactor them into presenters and viewmodels and keep the View as thin as possible. So, I would just use ViewModel in BindingListView, not the interface IViewModel.
So the BindingListView<ConcreteViewModel> will naturally make sense to me that it doesn't accept a model Interface.
Refer to this question about MVVM MVP design and unit testing WinForm:
Should I unit-test my view in MVP(or VM) or how to keep the code in the view to a minimum?
Answer: Why are they creating a new method get for properties, why? Couldn't they just use that PropertyInfo and get the property value? If I were making this I would have interfaces in mind and not restrict users from using them. They are creating same kind of method that calls original "get" method, I don't understand what is the point of this.
 private static GetPropertyDelegate BuildGetPropertyMethod(PropertyInfo pi)
        {
            MethodInfo getMethod = pi.GetGetMethod();
            Debug.Assert(getMethod != null);

            DynamicMethod dm = new DynamicMethod("__blw_get_" + pi.Name, typeof(object), new Type[] { typeof(T) }, typeof(T), true);
            ILGenerator il = dm.GetILGenerator();

            il.Emit(OpCodes.Ldarg_0);
            il.EmitCall(OpCodes.Call, getMethod, null);

            // Return the result of the comparison.
            il.Emit(OpCodes.Ret);

            // Create the delegate pointing at the dynamic method.
            return (GetPropertyDelegate)dm.CreateDelegate(typeof(GetPropertyDelegate));
        }

Fixed for sorting:
            private static Comparison<T> BuildComparison(string propertyName, ListSortDirection direction)
        {
            PropertyInfo pi = typeof(T).GetProperty(propertyName);
            Debug.Assert(pi != null, string.Format("Property '{0}' is not a member of type '{1}'", propertyName, typeof(T).FullName));

            if (typeof(IComparable).IsAssignableFrom(pi.PropertyType))
            {
                if (pi.PropertyType.IsValueType)
                {
                    return BuildValueTypeComparison(pi, direction);
                }
                else
                {
                    //CHANGED!!!!!
                    //GetPropertyDelegate getProperty = BuildGetPropertyMethod(pi);
                    return delegate(T x, T y)
                    {
                        int result;
                        //CHANGED!!!!!
                        object value1 = pi.GetValue(x, null);// getProperty(x);
                        //CHANGED!!!!!
                        object value2 = pi.GetValue(y, null); //getProperty(y);
                        if (value1 != null && value2 != null)
                        {
                            result = (value1 as IComparable).CompareTo(value2);
                        }
                        else if (value1 == null && value2 != null)
                        {
                            result = -1;
                        }
                        else if (value1 != null && value2 == null)
                        {
                            result = 1;
                        }
                        else
                        {
                            result = 0;
                        }

                        if (direction == ListSortDirection.Descending)
                        {
                            result *= -1;
                        }
                        return result;
                    };
                }
            }
            else if (pi.PropertyType.IsGenericType && pi.PropertyType.GetGenericTypeDefinition().Equals(typeof(Nullable<>)))
            {
                var compare = typeof(Nullable).GetMethod("Compare", BindingFlags.Static | BindingFlags.Public).MakeGenericMethod(pi.PropertyType.GetGenericArguments()[0]);
                return delegate (T x, T y)
                {
                    return (int)compare.Invoke(x,new[] { pi.GetValue(x, null), pi.GetValue(y, null) } );
                };
                //return BuildNullableComparison(pi, direction);
            }
            else
            {
                return delegate(T o1, T o2)
                {
                    if (o1.Equals(o2))
                    {
                        return 0;
                    }
                    else
                    {
                        return o1.ToString().CompareTo(o2.ToString());
                    }
                };
            }
        }


Salesforce query to get task's comment

Question: I am trying to get the Comments from Salesforce tasks through a SOQL query like:
<code>SELECT Id, Comments from Tasks
</code>
but it says "Comments" is not an existing field.
What should I use?
Answer: The field name of the Comments is "Description".
Change the query as below it will work.
"SELECT Id, Description from Task"
Regards,
Naveen
Software Engineer
http://www.autorabit.com

Salesforce query to get task's comment


Finding when a sum is convergant

Question: I want to show for which values of $p$ the series $\sum_{n=1}^\infty \frac{1}{n^p}$ converges/diverges using only Raabe's test.
I tried to check but I always got the following result which ended up not helping at all:
$L=lim_{n\to\infty} n(1-\frac{\frac{1}{n^{p+1}}}{\frac{1}{n^{p}}})=lim_{n\to\infty} n(1-\frac{n^p}{n^{p+1}})=lim_{n\to\infty} n(1-\frac{1}{n})$
And that doesn't even have $p$ in it!
What should I do?
Comment: The variable of the serie is $n$, not $p$.
Comment: Oh you're right!
Answer: That means:
$L=lim_{n\to\infty} n(1-\frac{\frac{1}{(n+1)^p}}{\frac{1}{n^{p}}})=lim_{n\to\infty} n(1-\frac{n^p}{(n+1)^p})=lim_{n\to\infty} n(1-(\frac{n}{n+1})^p)$
And then what?

Finding when a sum is convergant


Creating Subdomains with Apache

Question: I've got a server running Ubuntu 10.10 and LAMP that I'd like to configure a subdomain in Apache for. I have two websites that I want to serve:
mydomain.com (which resides in /var/www/mydomain)
AND
clients.mydomain.com (which resides in /var/www/clients)
I'm trying to use name-based virtual hosts and can't get the subdomain (clients.mydomain.com) working but mydomain.com resolves just fine.
Here's my configuration:
/etc/apache2/sites-enabled/mydomain.com.conf
<VirtualHost *:80>
DocumentRoot "/var/www/domain"
ServerName mydomain.com
<Directory "/var/www/mydomain">
allow from all
Options +Indexes
</Directory>
</VirtualHost>

/etc/apache2/sites-enabled/clients.mydomain.com.conf
<VirtualHost *:80>
DocumentRoot "/var/www/clients"
ServerName clients.mydomain.com
<Directory "/var/www/clients">
allow from all
Options +Indexes
</Directory>
</VirtualHost>

I also have an A record pointing clients.mydomain.com to the Ip of my server. Any idea on what's wrong?
Comment: Post the output of `/usr/sbin/apache2ctl -S`
Comment: You mentioned below it was a DNS issue. Tip: when you lower records, change the TTL beforehand. And, I prefer to keep them on 3600 sec (an hour) normally anyway. Another thing is, don't access the address until you've made the record, otherwise all servers in the chain will cache the non-record for days. And, you can run "dig A sub.domain.com @nameserver" to see if the record exists. And if you want to work without waiting for DNS, add "18.104.22.168 sub.domain.com" to your hosts file.
Answer: You point to sites-available. Are these sites symlinked in /etc/apache2/sites-enabled/? You can enable them with a2ensite.
If that's not it, do you have a NameVirtualHost *:80 in /etc/apache2/ports.conf? It's Ubuntu, so probably.
What does apachectl -t say?
Comment: They are symlinked in sites-enabled (sorry for the confusion). I have NameVirtualHost *:80 in ports.conf. apachectl -t and apache2ctl -t say command not found. Are they not compatible with bash?
Comment: You don't have apace2ctl? What does aptitude search "^apache2" say? Are you running it as root?
Answer: Turned out to be a DNS propagation issue. That A record took nearly 48 hours to spread around.
Comment: 48 hour propagation is normal! In fact count on it taking up to 72 hours.

Creating Subdomains with Apache


Propositional Logic: Validity of sequent $\lnot\Phi_1 \land \lnot \Phi_2 \vdash \Phi_1 \rightarrow \Phi_2$

Question:
Propositional Logic: Validity of sequent $\lnot\Phi_1 \land \lnot \Phi_2 \vdash \Phi_1 \rightarrow \Phi_2$

$\lnot \Phi_1 \land \lnot \Phi_2$ (premise)
$\Phi_1$ (assumption)
$\lnot \Phi_1$ ($\land e_1$) 1
$\bot$ ($\lnot e$) 2,3
$\Phi_2$ ($\bot e$) 4
$\Phi_1 \rightarrow \Phi_2$ ($\rightarrow i$) 2,5

All this look really weird to me. I work with $\Phi_1$ and $\lnot \Phi_1$ at the same time.
Can someone tell me how to proceed and why this is valid?

Taken from the book "Logic in Computer Science" by Michael Huth and Mark Ryan.
Comment: The question is a strange one - are you sure you've stated it correctly? The deduction theorem lets you rephrase as $\{\neg \phi_1 \wedge \neg \phi_2  , \phi_1 \} \vdash \phi_2$, which is true by explosion.
Comment: I'm sure. Check the image I've uploaded on the book I'm working on. I'm trying to get into logic programming by reading this book.
Answer: This is correct.
If we already know $\neg \Phi_1$ holds then the implication $\Phi_1\rightarrow \Phi_2$ is just saying that "If $\Phi_1$ is also true (i.e. both true and false) then we may conclude anything, such as $\Phi_2$".
The whole statement is all about the principle of explosion ($\bot \vdash P$ for any $P$) and contradiction.
You can finnish the proof by using that we have a contradiction, deduce whatever you want hence conclude $\Phi_2$ and complete the proof by using $\rightarrow$ introduction.
Comment: Thanks @OveAhlman. Would you mind looking at my updated solution? Why do the premise include $\lnot \Phi_2$ - I don't use it for anything. Have I made a mistake?
Comment: @NicolasLykkeIversen The updated solution is correct. As you point out, $\neg \Phi_2$ is not used for anything, and I think it is included in the exercise just to make the exercise more perplexing and look even more contradictory.
Comment: Thank you for your assistance @OveAhlman.


Authorize create action based on the parent_id of the object

Question: I have two models other than User model. Hotel and Room.
Here is the scenario:
hotel :has_many => :rooms

A User can create rooms if the associated hotel is assigned to the user. How can I implement authorization for the create room action in ability, as hotel_id is unknown till the room is going to be saved?
I can implement a can? method in create action under the rooms controller , but I am expecting to have it in ability. This problem should be resolved in nested_resources pattern but somehow I have to create rooms in a separate controller (not nested).
Answer: You can use the nested_resources facility as outlined here: https://github.com/ryanb/cancan/wiki/Nested-Resources but only set this for the specific create action of the separate controller using the specific action authorising methods outlined here: https://github.com/ryanb/cancan/wiki/authorizing-controller-actions
Edit - Updated with example for ability.rb as requested in comments:
can :create, Room, :hotel => { :user_id => user.id }

Comment: yes!, As I mentioned in the post, using nested resources we can achieve that. As I have to use separate controller I used the second approach. That is I've put a authorization check on controller. I was expecting something to be manageable through the ability module. I guess there is no way for that
Comment: I've updated with the standard way of doing such abilities in the Ability.rb file


What causes viewflow to throw 'More than one join instance for process found'

Question: I have a django-viewflow workflow which includes a 3-way Split() to handle email, SMS, etc. Since each of those activities may take an extended period of time to complete, I represent each one of the 3 split branches as a pair of nodes:

A normal Handler() node which spawns a Celery job.
A custom wait-for-celery node.

The custom node looks like this:
class CeleryEvent(mixins.TaskDescriptionViewMixin,
                  mixins.NextNodeMixin, mixins.DetailViewMixin,
                  mixins.UndoViewMixin,
                  mixins.CancelViewMixin, Event):
    ....
    activation_class = derived-from-AbstractJobActivation
    setting task_type = "somestring"

The invocation of the Viewflow code on Celery job completion follows the model of another question, and specifically includes the lock incorporated there. Generally, the result works fine. However, once in a while,I get this exception from Viewflow 1.3.0's join.py:
tasks = flow_class.task_class._default_manager.filter(
        flow_task=flow_task,
        process=process,
        status=STATUS.STARTED)

if len(tasks) > 1:
    raise FlowRuntimeError('More than one join instance for process found')

The 3 branches join like this:
close_join = flow.Join(wait_all=True). \
    Next(this.alert_devops)

I'm a bit baffled as to the cause of this as on inspection after the error, the combination of process and flow_task for the close_join in state STARTED does occur twice. I'm wondering if something I am doing could possibly be causing the issue. As far as I know, none of my code actually writes to this table directly.
I do note that the Task table does not have a unique_together('process', 'flow_task'), which I think might be because a Viewflow loop would cause the same flow_task to be hit multiple times. Since my code does not have loops (yet), I wonder if it would be a good idea to temporarily add such a constraint; at least then the creator of the illegal state would be the point of failure?
Is it possible that the lock being taken is not safe across processes? Since Celery is running this piece of code on several processes in the machine, that might then explain the problem?
    lock = self.flow_class.lock_impl(self.flow_class.instance)
    with lock(self.flow_class, task.process_id):
        #
        # Re-acquire the task under a lock (see the StackOverflow thread).
        #
        task = self.flow_class.task_class._default_manager.get(pk=task.pk)
        activation = self.activation_class()
        activation.initialize(self, task)
        activation.start()
        activation.done()

Comment: I just realised that I was lookingat my own tasks in the Task table. If I look at the "Join" node called "close_join" which comes after the t3 branches join togther, then yes, there are indeed two of them in the STARTED state.
Answer: I believe the lack of locking is indeed the issue. As per the docs:

Locking is not enabled by default. You need to choose proper lock implementation and enable it.


Limit of $\sin(1/x)$ - why there is no limit?

Question: $$ \lim_{x\to 0+} \sin\left(\frac{1}{x}\right)$$
I know that there is no limit.
but, why there is no limit?
I tried $x=0.4$, $x=0.3$, $x=0.1$, it looks like the limit is $0$.
And how can I show that there is no limit? I tried to calculate it like all the other functions, and I got wrong result and I don't know why:
$$\lim_{x \to 0+} \sin\left(\frac{1}{x}\right) = \sin\left(\frac{1}{0^+}\right) =  \sin\left(\frac{1}{\infty}\right) = \sin(0) = 0.$$
Comment: Since when is $0^+=\infty$?
Comment: How come $\frac{1}{0+}=1/\infty$?Did you get where you went wrong?
Comment: Try taking the limit along the sequence $x_k = \frac{1}{k \pi/2}$ with $k$ integer.
Comment: You cannot substitute $0^+$ and $\infty$ as if they were numbers, because they actually aren't.
Comment: I would be highly surprised if this question isn't a duplicate.
Answer: Why there is no limit?
The graphic can help you understand why and suggest you some approach for the proof:

Remark: You have to be careful with tables of values because they can be misleading:
\begin{array}{ c | c c c c }
x &  \frac{1}{2\pi} & \frac{1}{3\pi} & \frac{1}{4\pi} &\frac{1}{5\pi} \\ \hline
 \sin\left(\frac{1}{x}\right) & 0     & 0   & 0     & 0    \\
                        \end{array}
\begin{array}{ c | c c c c }
x &  \frac{2}{5\pi} & \frac{2}{9\pi} & \frac{2}{13\pi} &\frac{2}{17\pi} \\ \hline
 \sin\left(\frac{1}{x}\right) & 1     & 1   & 1     & 1    \\
                        \end{array}
(The tables above are a sketch of the proof - see Theorem 2.4 here.)
Comment: Thank you for the nice explanation! May I ask what's the title of the book you linked? I'd be interested in the other chapters of it. Edit: Never mind, I found it here: https://www.math.ucdavis.edu/~hunter/m125a/
Answer: That's because your limit is equivalent to $$\lim_{x\to\infty} \sin(x)$$
and $\sin$ is oscillating when $x\to\infty$.
Answer: Hint: $\frac1x$ has a right-sided limit of $+\infty$. Does $\sin x$ have a limit for $x\to\infty$?
Answer: You made an error.
\begin{array}{ c | c c c c }
x &  .4 & .3 & .2 &.1 \\ \hline
 \sin\left(\frac{1}{x}\right) & 0.5984721     & -0.190568   & -0.9589243     & -0.54402113    \\
                        \end{array}
This does not suggest that the limit approaches $0$.
Second, you mean to say
$$\lim_{x\to 0^+} = \sin\left(\frac{1}{x}\right) = \sin(\infty)$$
not $\lim_{x\to 0^+}\sin\left(\frac{1}{x}\right) = \sin\left(\frac{1}{\infty}\right).$
Recall that $$\lim_{x\to 0^+} \frac{1}{x} = \infty.$$
Answer: Consider the two subsequences $x_k = \frac1{(4k + 1)\pi/2}$ and $x'_k = \frac1{k\pi}$ with $k \in \mathbb Z$. We have that for $k \to +\infty$, $x_k,x'_k \to 0^+$. Clearly,
$$\begin{align*}
&\lim_{k \to +\infty} \sin\left(\frac1{x_k}\right) = 1\\
&\lim_{k \to +\infty} \sin\left(\frac1{x'_k}\right) = 0
\end{align*}$$
and therefore the limit $x \to 0^+$ does not exist. We used the theorem that states that if a sequence converges, then every subsequence converges to the same limit. By modus tollens, our sequence does not converge.
More info about the theorem here:
Prove: If a sequence converges, then every subsequence converges to the same limit.
Answer: Lets look at $sin(\frac{1}{x}) $ function,

Now you can get a idea how it behaves near to 0. That is oscillating near 0. THat's why we can't get limit when x goes to 0 of $sin(\frac{1}{x}) $ function. So let's move into it's proof,
Have to prove $\lim_{x \to 0+} sin(\frac{1}{x})=Does  \: not \: exist$
$$\forall \varepsilon >0 \;  \exists \delta \; s.t \;  0< x  < \delta \mapsto |sin\frac{1}{x}-L|<\varepsilon$$
Let $\varepsilon=\frac{1}{2}$,
$$0< x   < \delta \mapsto |sin\frac{1}{x}-L|<\frac{1}{2}$$
Assume that, $$\lim_{x \to 0+} sin(\frac{1}{x})=L \; \in \mathbb{R}$$
Let $$x_{1}=\frac{1}{2n\pi+\frac{\pi}{2}} \; \;  (n\in Z^{+})\; \; and \; \; x_{2}=\frac{1}{2n\pi+\frac{3\pi}{2} } \; \;  (n\in Z^{+})$$
So we have to get range of n,
$$0<\frac{1}{2n\pi +\frac{3\pi}{2}} < \frac{1}{2n\pi+\frac{\pi}{2} }<\delta$$
$$\frac{1}{2n\pi+\frac{\pi}{2} }<\delta$$
So we get,
$$\frac{1}{2\pi\delta }-\frac{1}{4}<n\; \;  (n\in Z^{+})$$
Now,
$$|sin(x_{1})-L|=|1-L|<\frac{1}{2}\Rightarrow \mathbf{C}$$
$$|sin(x_{2})-L|=|-1-L|=|1+L|<\frac{1}{2}\Rightarrow \mathbf{D}$$
By C + D we get,
$$|1-L|+|1+L|< 1$$
$$|(1-L)+(1+L)|\leq |1-L|+|1+L|< 1$$
$$2\leq |1-L|+|1+L|< 1$$
$$2<1 \; \;\; \; (\therefore contradiction)$$
Assumption is not true. $\: \: \therefore\:\lim_{x \to 0+} sin(\frac{1}{x})$ has no limit.
Comment: Nice proof, but shouldn't it be $|\sin(\frac{1}{x_{1}})-L|=|1-L|<\frac{1}{2}$ and $|\sin(\frac{1}{x_{2}})-L|=|1+L|<\frac{1}{2}$? If not, please explain.


case statement to prevent opening of certain files?

Question: I have a Dir of files; most are unique, but some are variations of others e.g. Hires Month 1, Hires Month 2 etc.... I want to skip over these when opening files, so I made a Case statement with a like operator, but it doesn't seem to be working.
    Dim BasePath As String: BasePath = "\\All Data\" & strFolderName & "\"
Dim filename As String: filename = Dir(BasePath & "\*.xlsx")
Dim wb As Workbook
    i = 0
    Do While filename <> ""
        Select Case True
            Case filename Like "Store*"
            Case filename Like "Hires Month*"
            Case filename Like "Current - Store*"
            Case filename Like "Terms Month*"
            Case Else
            Set wb = Workbooks.Open(BasePath & "\" & filename)
            i = i + 1
                With wb.Worksheets("Sheet1")
                    .UsedRange.Copy Destination:=wbTemplate.Worksheets(varWsName(i)).Cells(1, 1)
                End With
            wb.Close
            filename = Dir
        End Select
    Loop

What could be causing the logic not to apply when supposedly triggered?
Comment: Does the like only apply to the first? or are these getting skipped too
Answer: Turn it round, thus
Select Case True
       Case filename Like "Store*", filename Like "Hires Month*", filename Like "Current - Store*", filename Like "Terms Month*"

Comment: @NidenK Try moving `filename = Dir` outside the Select Case.
Comment: So when I do this, it triggers the first instance of `Store *.xlsx`, and skips to `Loop`, but it never opens another file....its stuck on the first Case statement file


Should CPU usage be restricted?

Question: I have a computation-intensive application, which needs to run on a Windows server with no other applications on it. The application is designed for horizontal scalability so that it can run on multiple servers if the input load is more. Should I be worried about the CPU usage and ensure that every time it goes over a certain threshold, I should bring in a new server and get the application running on it to spread the load? Or is it ok if the app runs continuously at 100% CPU load?
Basically are there any disadvantages of letting an app run at 100%?
I understand overheating of the CPU could be an issue.
Also context switching between threads of the application could cause the overall thorughput to be reduced.
Any other? Is there any guideline regarding a threshold to be set for CPU utilization?
Thanks,
Yash
Comment: Rather than being worried about how you can run at 100% CPU all the time, the question should be why the application consumes that much CPU constantly?
Comment: http://superuser.com/questions/431309/what-is-the-impact-of-running-the-cpu-at-100-for-long-periods-at-a-time
Answer: Intel (i) series i5 and i7 allow hyper-threading and turbo-boosting which makes the CPU run at 105-110% from my knowledge.
Overheating will be a problem but if you can set affinities for different cores on different times if you have a quad core 2 can run on turbo-boost while the other two cool down. Assuming you can do that.
I hope I answered your question (in a way).

Should CPU usage be restricted?


Is there ever a reason to use tmpfs instead of or in addition to zRAM?

Question: I'm thinking about using zRam on my server.
From what I've read about zRam, it apparently does everything tmpfs does and more.
Once zRAM is working, is there any reason to keep tmpfs around?
Is there anything tmpfs does that zRAM can't do?
Is there ever a reason to use tmpfs instead of or in addition to zRAM?
Answer: Data written to a tmpfs will be kept in-memory uncompressed as long as enough memory is available. Once the system comes under memory pressure, it tries to move data from the tmpfs (and from any other places, like normal process memory) to any swap (including swap on zram) it can find. Also you can change parameters of a tmpfs by doing a simple mount -o remount, especially the size of the file system or the number of inodes.
Using zram as the backing of a temporary file system has the disadvantage, that all data written there needs to be compressed immediatelly (and decompressed upon use). Also you need to put a file system on top, whiches structures need to be stored even if there are no actual files stored. Also you get limitations like max number of inodes, you can't easily change it's size and so on.
IMHO, best solution is to mount tmpfs somewhere to use and create a zram swap. It combines the advantages of both by minimizing the disadvantages.
Or buy more ram of couse ;)
Answer: You will get better performance from using zram for swap due to the compression that is possible.
zram-init makes zram simple to use. Remember to read the instructions.
Your kernel needs to be built with CONFIG_ZRAM_LZ4_COMPRESS enabled to be able to benefit from lz4 compression.
I use zram for swap & for /tmp (zram-init uses sensible default mount options for /tmp):
root@alpine [~]# cat /proc/swaps
Filename                Type        Size    Used    Priority
/dev/vda7                               partition   1045500 0   -1
/dev/zram0                              partition   1048572 0   16383
root@alpine [~]# mount|grep zram
/dev/zram1 on /tmp type ext4 (rw,nosuid,nodev,noexec,discard)

There are still use cases for using tmpfs. I use these for bind mounting volumes with secure mount options into lxc containers:
root@http [~]# mount|grep tmpfs
tmpfs on /var/www/domain.com/bin type tmpfs (ro,nosuid,nodev,relatime,size=200k)
tmpfs on /var/www/domain.comm/dev type tmpfs (rw,nosuid,relatime,size=4k,mode=755)

Comment: What you say is true. But I don't see how it's relevant to my question about tmpfs. Did you perhaps mean to post this as an answer to some other question?
Comment: Is "There is never any reason to use tmpfs. Instead, zram is just as easy to use and gives better performance" a valid summary? Because I'm not seeing any reasons to use tmpfs in your answer. (It may be obvious to you the reasons why using tmpfs is a great idea "for bind mounting volumes with secure mount options into lxc containers", but I wish you would tell me what those reasons are. Or is there no actual reason, and you're still doing it that way only because you haven't had time to convert that system zram?)
Comment: perhaps you should try actually using them both.
Comment: @David Cary - answer expanded


What is the difference between the classification and the pattern recognition?

Question: Could somebody explain this difference just as simple as possible?
Answer: Pattern recognition is the "automated discovery of patterns in a training set", and so it is a general term for machine learning.
Classification is the supervised learning problem whose target value is a finite set of classes (as opposed to regression, wherein the target value is a continuous variable).
Therefore, classification is a kind of pattern recognition problem.
Comment: well stated: I had not considered `pattern recognition` to be nearly synonymous with ML
Comment: @javadba To be honest, I lifted that from Bishop's introduction in "Pattern recognition and machine learning".
Answer: Classification is simply a more general term than pattern recognition. In both cases, you have a set of classes $K$ and a collection of observations, where each observation is represented by a set of features. Your job is to find a mapping of features to a member of $K$ such that you minimize some measure/estimate of out-of-sample classification error.
In pattern recognition, you are simply using a very complex/large feature space. For example, facial recognition will have an input space equal to the number of pixels in the image. This is no different than classifying a loan application as high or low risk based on several measures of creditworthiness.
Comment: Pattern recognition is a more general term.


When can one use a non-stationary VAR validly?

Question: In Chris Brooks book Introductory Econometrics for Finance on page 292 when discussing the problem with VARS he mentions.

... many proponents of the VAR approach recommend that differencing to induce stationarity should not be done. They would argue that the purpose of VAR estimation is purely to examine the relationship between the variables, and that differencing will throw away information on any long-run relationship between the series away..

Who are these "proponents of the VAR approach" and under what circumstances can I use a Non-stationary VAR?
Answer: I am guessing that the thrown away information the author is referring to is alluding to the situation where the VAR model is cointegrated. This is one situation, but there might be others.
Start with a non-stationary VAR(2)
$$
Y_t = A_1 Y_{t-1} + A_2 Y_{t-2} + U_t.
$$
Then difference once to get
\begin{align*}
\Delta Y_t &= -IY_{t-1} + A_1 Y_{t-1} + A_2 Y_{t-2} + U_t \\
 &= -IY_{t-1} + A_1 Y_{t-1} + A_2 Y_{t-1} - A_2 Y_{t-1} + A_2 Y_{t-2} + U_t\\
 &= -(I - A_1 - A_2)Y_{t-1} - A_2 \Delta Y_{t-1} + W_t \\
 &= \alpha \beta ' Y_{t-1} + \Gamma_1 \Delta Y_{t-1} + U_t.
\end{align*}
$(I - A_1 - A_2) = \alpha \beta '$ is singular because the determinant AR polynomial has a unit root. If you differenced $Y_t$ and estimated a VAR(1), you would be ignoring/omitting the $\alpha \beta ' Y_{t-1}$ term which describes an economic equilibrium between some of the elements of $Y_t$.
I'm looking at the table of contents for this book online right now, and this sort of stuff is discussed a bit later on. Keep reading; it looks like it will be in chapter 8, if you have the same edition.
Comment: Im using the second edition, which one are you using?
Comment: @EconJohn I don't have the textbook. I just searched online for it to see what it covered


Utility Word Problem

Question: You derive utility from Leisure (L) and your grade (G).  The higher they are, the happier you are.  Let your utility function be U(L,G).   Your grade is increasing in your study time (S).  You can therefore write G = f(S) where f is an increasing function describing how your grade relates to your investment in studying. It is the last day before the final exam and you have 24 hours to allocate between leisure and study.  Write the optimization problem you are facing.  You DO NOT need to solve it.
I'm confused with the equation I need to write.  This is for a ECO micro class.  Would the correct formula be:
max(L,G)
Subject to G = f(s)
Any and all help would be greatly appreciated.  Thanks in advance.
Answer: The time constraint is $L+S=24$ (you can spend time on leisure or studying).
Thus the problem is
$$\max_{L,G} u (L, G) \\
\mbox{s.t. }\> L+S=24 \\
\> G= f(S)$$
Comment: That makes sense.  Thank you.

Utility Word Problem


TCP SYN cookies are always turned on when enabled?

Question: For Linux kernel  2.6.32 if I set net.ipv4.tcp_syncookies = 1 will that be always used or only during a SYN flood attack?
I found 2 sources which say the opposite.
1:

"There are, however, two caveats that take effect when SYN cookies are in use. Firstly, the server is limited to only 8 unique MSS values, as that is all that can be encoded in 3 bits. Secondly, the server must reject all TCP options (such as large windows or timestamps), because the server discards the SYN queue entry where that information would otherwise be stored. [1]
While these restrictions necessarily lead to a sub-optimal experience, their effect is rarely noticed by clients because they are only applied when under attack. In such a situation, the loss of the TCP options in order to save the connection is usually considered to be a reasonable compromise."

2:

"The downside is that not all TCP data can fit into the 32-bit Sequence Number field, so some TCP options required for high performance might be disabled." This means that options such as selective ACKs and TCP Window Scaling won't work if you turn on SYN Cookies, even if your server isn't currently under attack."
Comment: I think it would be a good idea to mention where you got those two quotes from.
Comment: Those two sources don't say the opposite. One says syn cookies are only sent when under attack. The other says that enabling syn cookies prevents selective ACK and TCP window scaling from working even when the server isn't under attack. Quite possibly, it's precisely because what the first source says is true that the second source needs to point out that this happens regardless of whether the server is under attack or not. (If that never made any difference, why point out specifically in this case that it doesn't?)
Answer: The second source

"this means that options such as selective ACKs and TCP Window Scaling
won't work if you turn on SYN Cookies, even if your server isn't
currently under attack"

is simply bullshit, until

"TCP: request_sock_TCP: Possible SYN flooding on port 53. Sending
cookies. Check SNMP counters"

happens there is nothing disabled.
This is easy provable by comparing your performance with iperf3 and enabling/disabling "net.ipv4.tcp_sack" and "net.ipv4.tcp_window_scaling" while keep "net.ipv4.tcp_syncookies = 1" enabled.
Why do you read random sources instead the official ones?
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
tcp_syncookies - BOOLEAN
    Only valid when the kernel was compiled with CONFIG_SYN_COOKIES
    Send out syncookies when the syn backlog queue of a socket
    overflows. This is to prevent against the common 'SYN flood attack'
    Default: 1

    Note, that syncookies is fallback facility.
    It MUST NOT be used to help highly loaded servers to stand
    against legal connection rate. If you see SYN flood warnings
    in your logs, but investigation shows that they occur
    because of overload with legal connections, you should tune
    another parameters until this warning disappear.
    See: tcp_max_syn_backlog, tcp_synack_retries, tcp_abort_on_overflow.

    syncookies seriously violate TCP protocol, do not allow
    to use TCP extensions, can result in serious degradation
    of some services (f.e. SMTP relaying), visible not by you,
    but your clients and relays, contacting you. While you see
    SYN flood warnings in logs not being really flooded, your server
    is seriously misconfigured.


Merge a Git fork without changing master branch

Question: My up-to-date branch is master (commit M). I had to create the branch old from an old commit (commit O1) and make some commits on old (commit O3). Now, I want to merge old on master to save it, but I want to keep the last commit on master as it is right now (commit M).
How can I do this?
Comment: "I want to create a merge commit without creating any commits. How?" Sorry, no way.
Comment: https://git-scm.com/book/en/v2/Git-Tools-Replace may help.
Answer: There are two answers to that :

in git, a commit is identified by a hash, which will change if (among others) its parents change.

With this point of view : if you introduce something that merges together master and old, you can't have the same hash as your current commit on master, you will always create a new commit object.

regarding a commit's content, however, you can choose to store whatever content you see fit.

With this in mind : you can create a merge commit (one which will have two parents : your current master and old), but which has the exact same content as your current master.
There actually is an option to git merge to do exactly that :
# from branch 'master' :
git checkout master

# create a merge commit, but ignore all changes that come from 'old' :
git merge -s ours old

Here is a link to the list of built-in merge strategies, here is the section about ours :

ours
This resolves any number of heads, but the resulting tree of the merge is always that of the current branch head, effectively ignoring all changes from all other branches. It is meant to be used to supersede old development history of side branches. Note that this is different from the -Xours option to the recursive merge strategy.

the Examples section contains :

Merge branch obsolete into the current branch, using ours merge strategy:

$ git merge -s ours obsolete


atoi function causing segmentation (core dump) error

Question: I am trying to recreate the bash "head" and "tail" commands (which print the contents of files in different ways) using c programs. I figured that the getopt function would be useful for parsing the command-line options from the files to be printed. Here's what I have so far:
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>

const int BUFFER_SIZE = 100;

int main(int argc, char *argv[]) {
    int c = 10;
    int n = 0;
    int opt;
    while ((opt = getopt(argc, argv, "cn:")) != -1) {
        switch(opt) {
        case 'c':
            c = atoi(optarg);
            break;
        case 'n':
            n = atoi(optarg);
            break;
        }
    }
    printf("c = %d & n = %d\n", c, n);
}

The program compiles. However, when I run the executable with -c 5 as its command-line arguments, the program crashes. I don't understand what the problem is.
Comment: Use a debugger to look at the input to `atoi`.
Answer: Your optstring ("cn:") says that c does not take an argument, so in this block:
case 'c':
    c = atoi(optarg);
    printf("worked\n");
    break;

optarg is NULL, which is why atoi(optarg) leads to a segmentation fault.
If option -c takes an argument, then you want:
while ((opt = getopt(argc, argv, "c:n:")) != -1) {


É possivel codificar linha por linha em base64?

Question: Tenho um código que pega arquivos binários e os converte em base64, porém existem arquivos muito grandes e isto acaba usando muito da memória da máquina (e até mesmo do processo). Para ler arquivos grandes geralmente (sem convertê-los) eu faço isto:
$handle = fopen('ARQUIVO', 'rb');

while (false === feof($handle)) {
    $data = fgets($handle);
    ...
}

fclose($handle);

Assim evito exceder a memória e outros problemas do gênero. Mas o problemas é que com base64_encode eu necessito dos dados completos (a string inteira).
Como codificar arquivos binários grandes sem perder o desempenho?
Comment: Sim.
[Veja:](http://stackoverflow.com/questions/7920780/is-it-possible-to-base64-encode-a-file-in-chunks) Mas seus pedaços tem que ser em tamanhos múltiplos de 3 em questão de bytes.
Answer: É possível sim, o código abaixo foi retirado de um comentário sobre a função base64_encode.
$fh = fopen('Input-File', 'rb');
//$fh2 = fopen('Output-File', 'wb');

$cache = '';
$eof = false;

while (1) {
    if (!$eof) {
        if (!feof($fh)) {
            $row = fgets($fh, 4096);
        } else {
            $row = '';
            $eof = true;
        }
    }

    if ($cache !== '')
        $row = $cache.$row;
    elseif ($eof)
        break;

    $b64 = base64_encode($row);
    $put = '';

    if (strlen($b64) < 76) {
        if ($eof) {
            $put = $b64."\n";
            $cache = '';
        } else {
            $cache = $row;
        }

    } elseif (strlen($b64) > 76) {
        do {
            $put .= substr($b64, 0, 76)."\n";
            $b64 = substr($b64, 76);
        } while (strlen($b64) > 76);

        $cache = base64_decode($b64);

    } else {
        if (!$eof && $b64{75} == '=') {
            $cache = $row;
        } else {
            $put = $b64."\n";
            $cache = '';
        }
    }

    if ($put !== '') {
        echo $put;
        //fputs($fh2, $put);
        //fputs($fh2, base64_decode($put));
    }
}

//fclose($fh2);
fclose($fh);

Comment: Só uma nota: Alterei o tamanho de 4096 para 32 `$row = fgets($fh, 32);` e utilizei `memory_get_usage()` e `memory_get_peak_usage()` com uma image de 2,75Mb, tive os seguintes resultados: Inicial: memory 224488 e
memory_peak 237032

/ Final:
memory 224792
memory_peak 241600