Intermittent SQL Insert Error

Question: I am working on a system where we do a monthly job of inserting about a million  records into a table. It has been working without any errors and generally continues to do so. Strangely it seems to hit a hump at some point. This month the records up until the 13th September at 21:15:12 for only one of many similar productIDs inserted fine but then suddenly stopped inserting for just productID 37.
The code which fails is:
INSERT INTO tblFooBar (productID, dDate, dTime) VALUES (37,'2017-09-13','21:15:12')

It gives this error:

Microsoft OLE DB Provider for SQL Server error '80040e07' Conversion
  failed when converting date and/or time from character string.

If I remove the productID field it enters without any problem:
INSERT INTO tblFooBar (dDate, dTime) VALUES ('2017-09-13','21:15:12')

Comment: Could you show the table's columns data types?
Comment: Then I'm inclined to say that what we're told is happening is impossible, and you've somehow managed to simplify the problem to the point where it can't be reproduced. Specifically, I can think of no way to make SQL Server fail and issue an error with one statement and not the other if `tblFooBar` is a plain table without any triggers and the statements are issued literally with `sqlcmd` or SSMS on the same connection, regardless of the values involved. Throw in parameters, client code, `SET LANGUAGE` options or other things and it's possible, but otherwise no, barring a very implausible bug.
Comment: Did you try with `13-09-2017`?
Comment: Use ISO-8601 date format.
Comment: Consider parameterized queries to avoid parsing errors. If you must use string literals, use ISO 8601 unseparated date format (YYYYMMDD). Not sure what the root cause here is if it works just by removing ProduceID (trigger maybe?).
Comment: Does the productId exist?
Comment: Thanks for all the comments. The column datatypes are: productID = int, dDate = date, dTime = time(7)
Comment: It is worth noting that this has worked for about a million records every month this year so the formats do match up. It is the randomness of the error that has me puzzled. Why would it insert ten million times but then randomly start failing for just one productID in the middle of a month? I just checked and that product is in the table 4,240,994 times already.
Comment: I'm willing to bet good money that (per Dan) there's a trigger on `tblFooBar` that ultimately causes the error, since the logic for parsing dates and times doesn't vary with what's in the other columns.
Comment: Yes, I also considered triggers but there are no stored procedures used in the database.
Comment: Triggers and stored procedures have nothing to do with each other. You can call a sproc from a trigger, but that's it. Triggers on tables are represented separately in SSMS, or check with `select * from sys.triggers`.
Comment: Hi Jeroen. Many thanks for your reply. There are no records in sys.triggers.
Comment: That's exactly why I am foxed. I have been scratching my head on this for days. To add to the simplicity tblFooBar is the only table in the database.
Comment: INSERT INTO tblFooBar (productID) VALUES (37) also inserts without error.
Comment: Any `CHECK` constraints? The final thing you can try is actually compare execution plans between the queries (Query -> Include Actual Execution Plan in Management Studio). They should both produce trivial insert plans.
Comment: There are no check constraints and 2017-09-13 worked for previous inserts.
Comment: I included the Actual Execution Plan and it only displayed for a test entry. It gave me (reading from left to right) INSERT cost 0%, CLUSTERED INDEX INSERT cost 100%, COMPUTE SCALAR cost 0%, COMPUTE SCALAR cost 0% and CONSTANT SCAN cost 0%
Comment: An update on this for anyone that is interested. I paired it down to basic principles as much as possible and have been validating the data with a convert function SELECT CONVERT(date,'2017-09-14'). When I run the native input it fails but when I overtype the 2017 in management studio in one go it executes successfully. If I overtype each number individually and try to execute it between each number it fails.
Comment: The plot thickens! When I copy a failed date that looks like '2017-09-14' from management studio into Textpad I get '?2017-09-14'.
Answer: I am glad to say I have resolved the issue. It was not really a SQL Server issue.
The problem was to do with the source dates that I was copying into the SQL statement. They had spurious ï, » and ¿ characters that a third party program had added to the dates and that I had not detected because of encoding.

Intermittent SQL Insert Error


Why is Kibana trends not working?

Question: I have created an data river from Twitter and loaded that data into Kibana dashboard using elasticsearch. While all the analytics tools mentioned there are working properly with my data,the "Trends" tool is showing an error " Oops! A time filter must exist for this panel to function". And I have tried applying filters too. But still the problem exist. Can anyone help me on this?
Answer: I also faced this exact issue. The problem was that kibana assumes the timestamp to be @timestamp field , you need to override this to created_at timefield.
This can be done by going to the global settings in kibana and change @timestamp to created_at field.
After this , you can first change the dashboard settings to "Last 30 minutes" or so and use the trends panel.

Why is Kibana trends not working?


Argument in phablet-flash for Nexus73G

Question: I trying install Ubuntu-touch on Nexus 7 3G, using Touch Install.
But on the step 4, I received an error:
~ $ sudo phablet-flash -d grouper -b
phablet-flash: error: argument : invalid choice: 'grouper' (choose from 'cdimage-touch', 'cdimage-legacy', 'ubuntu-system', 'community')

Comment: ~ $ sudo phablet-flash cdimage-touch
INFO:phablet-flash:Device detected as tilapia
ERROR:phablet-flash:Unsupported device, autodetect fails device
Comment: As of **March 2013** use of the **`phablet-flash`** tool has been deprecated, and you should use **`ubuntu-device-flash`**. Support (as reference hardware) for the **Nexus 7** (original model) is also now deprecated. *(see more below)*
Answer: As of version 1.0 (of phablet-tools) the syntax you need changes to:
phablet-flash cdimage-touch -d grouper -b
You should not need sudo for this command.

See: phablet-flash -h OR http://sergiusens.github.io/posts/updates-to-phablet-flash.html
Verify what version you have with: dpkg -s phablet-tools | grep Version
This new version definitely works for Nexus 7 and Nexus 10, but I have not yet tried with Nexus 7 3G.

UPDATE: (23-Aug-2013)
The ASUS Nexus 7 3G (and in particular the 32 GB model) are somewhat different from the base model of the Nexus 7 ('grouper'). They have support for GSM 3G/4G and other low-level firmware changes.
Before Ubuntu Touch was 'flipped' (still ran Ubuntu in a container inside Android), this did not provide an insurmountable problem. However, the additional features not found in grouper were also not available.
Now with the 'flipped' design, and with current changes to the underlying Ubuntu Touch OS and UI, only the base model Nexus 7 (grouper) is currently supported.
See also:
Latest two images will not install on Nexus7 (32GB + 3G), but older were Ok

The latest two (newest) daily images will not install on my Nexus 7, 32GB + 3G. The older versions work fine. Background looks like coloured snow. ..

Nexus 7 3G [2012] problem with ubuntu touch

Nexus 7 3G is not equivalent to grouper.
Nexus 7 is a commercial name that hides a lot of different hardware configurations in it.
If the hardware codename is tilapia [2012, 3G] or razor [2013] the device will not work without modifications or what people call ports, please refer to https://wiki.ubuntu.com/Touch/Devices

and more technical:
https://bugs.launchpad.net/ubuntu-nexus7/+bug/1079729

Ubuntu uninstallable on 32GB 3G Nexus 7
Nexus7 variants without 3G support map their user data partition (UDA) to mmcblk0p9, where variants with 3G support map UDA to mmcblk0p10 ..

UPDATE: (09-Mar-2014)
This question/answer is no longer current.

As of March 2013 use of the phablet-flash tool has been deprecated, and you should use ubuntu-device-flash.
Support (as reference hardware) for the Nexus 7 (original model) is also now deprecated, with development focused on: Nexus 4 smartphone, Nexus 7 (2013) tablet, and Nexus 10 tablet.
Support for Galaxy Nexus is "maintenance only" (major bug fixes).

Argument in phablet-flash for Nexus73G


Order By 2 columns

Question: I have this table and .
 id  |  item_id  | created_at
-----+-----------+------------
  1  |   Apple   | 2017-03-21
  2  |   Grape   | 2017-03-23
  3  |   Grape   | 2017-03-24
  4  |   Apple   | 2017-03-25

I want to order by created_at and also at the same time order by item_id like this:
 id  |  item_id  | created_at
-----+-----------+------------
  4  |   Apple   | 2017-03-25
  1  |   Apple   | 2017-03-21
  3  |   Grape   | 2017-03-24
  2  |   Grape   | 2017-03-23

So if i add a new row for item_id: Grape my new results should be like this:
 id  |  item_id  | created_at
-----+-----------+------------
  5  |   Grape   | 2017-03-28  (NEW)
  3  |   Grape   | 2017-03-24
  2  |   Grape   | 2017-03-23
  4  |   Apple   | 2017-03-25
  1  |   Apple   | 2017-03-21

and then if i add new row for item_id: Apple it should be like this:
 id  |  item_id  | created_at
-----+-----------+------------
  6  |   Apple   | 2017-03-28  (NEW)
  4  |   Apple   | 2017-03-25
  1  |   Apple   | 2017-03-21
  5  |   Grape   | 2017-03-27
  3  |   Grape   | 2017-03-24
  2  |   Grape   | 2017-03-23

...So it orders by the latest created_at and show the other rows with the same item_id below it
I have tried ORDER BY created_at, item_id  DESC but it does not works and give me this instead:
 id  |  item_id  | created_at
-----+-----------+------------
  6  |   Apple   | 2017-03-28
  5  |   Grape   | 2017-03-27
  4  |   Apple   | 2017-03-25
  3  |   Grape   | 2017-03-24
  2  |   Grape   | 2017-03-23
  1  |   Apple   | 2017-03-21

Comment: Don't you know that you are asking a multiple kind of databases. And expect that you got a multiple type of queries too. Not all queries are working with 3 different types of databases.
Comment: 1. group by item_id 2. get row_number() 3. make a nested query and select other columns, with sorting by row_number(), created_at desc
Comment: @zerkms do you have a full example query?
Comment: Just do it step by step, ask a particular question when stuck (ps: my advice is actual for ANSI SQL and postgresql, since mysql does not support window functions)
Comment: You can use variable in mysql.  I posted an answer which uses that.
Comment: Nevermind, I see the other mysql answer which uses just max and is the way to go.
Answer: You need to first identify your first rule which is get the max created_at per item_id, therefore the subquery with the item_id grouped and a row_number() added so it can know which fruit cames first. Then just join with your table.
SELECT tt.id, tt.item_id, tt.created_at
  FROM test_table tt
         INNER JOIN
      (SELECT item_id,
              MAX(created_at),
              ROW_NUMBER() OVER (ORDER BY MAX(created_at) DESC) rn
         FROM test_table
        GROUP BY item_id
        ORDER BY 3) ord
     ON tt.item_id = ord.item_id
 ORDER BY ord.rn, tt.created_at DESC;

Comment: Great, now this is the best looking answer, +1 :-)
Comment: Yes, I saw it. I was preparing the setup in my machine to test my solution and when I came back it was exactly what you suggested in comments. @zerkms
Comment: I think I've fixed it now. Tested it and it is working.
Comment: When i add a new row for Grape all the other Grape rows does not move to the top but instead all Apple rows are still at the top. I would like for the latest created_at Fruit to move to the top with all it's similar Fruit rows
Comment: I think I forgot to add the DESC on the order by in the subquery. Try it now please.
Comment: It's still the same result
Comment: You don't have sorting in the nested query, hence the `ROW_NUMBER()` is not ordered.
Answer: SQLFiddle: PostgreSQL
WITH grouped AS (
  select   item_id, max(created_at) as max_dt
  from     tbl
  group by item_id
)
SELECT     tbl.*
FROM       grouped
LEFT JOIN  tbl      USING (item_id)
ORDER BY   grouped.max_dt desc,
           grouped.item_id,      -- important if apple and grape both have same max dt
           tbl.created_at desc;

subquery to get the most recent dates by item (the item grouping)
left join those results on the actual table to get desired records
Answer: For mysql, try this:
select item.*
from item
left join (
    select item_id, max(created_at) maxdate
    from item
    group by item_id
) t on item.item_id = t.item_id
order by t.maxdate desc, item.created_at desc

Demo1 in SQLFiddle
Demo2 in SQLFiddle
Answer: Please try -
SELECT id,
       item_ID,
       created_at
FROM tblOrders
ORDER BY item_ID,
         item_ID = ( SELECT item_ID
                     FROM tblOrders
                     WHERE id = ( SELECT MAX( id ) FROM tblOrders ) ),
         created_at;

Comment: I will test this shortly in a similar database of mine, once I find one that is suitable and adapt the statement to suit.
Comment: Oh, boy was I way off.  I have changed my answer to something that I have tested and confirmed as working.
Answer: Here is a mysql version
set @ordr = 0;
select t.*
from tbl t
join (
    select o.item_id, o.created_at, @ordr := @ordr + 1 as `order`
    from (
        select tbl.item_id, max(tbl.created_at) as created_at
        from tbl
        group by tbl.item_id
        order by created_at desc
    ) as o
) sub
on t.item_id = sub.item_id
order by sub.`order`, sub.created_at desc

Order By 2 columns


Responsive/Fluid 4 column with fixed margin

Question: For the life of me I can't workout why my 4 col layout is breaking, could someone help me fix? When you make the browser smaller, the layout changes to 3 cols, I need to be 4 cols at all screen sizes.
Its a fluid width ('max-width: 1056px') with a '12px' fixed margin. http://jsfiddle.net/KwUcG/1/
I don't want to use '@media' and breakpoints, hence why I've used max-width and percentages.
HTML

Home to half a million objects of art and archaeology from pre-history to the present day.

Home to half a million objects of art and archaeology from pre-history to the present day.

Home to half a million objects of art and archaeology from pre-history to the present day.

Home to half a million objects of art and archaeology from pre-history to the present day.

CSS
.wrap {
padding: 0 40px;
margin: 0 auto;
background: #e4f;
overflow: hidden;
}

.inner {
    max-width: 1056px;
    margin: 0 auto;
    background: #34e;
    overflow: hidden;
}

.fourcol {
display: inline-block;
width: 24%;
margin-left: 12px;
background: #ccc;
}

.fourcol:first-child { margin-left: 0; }

Comment: For a fluid design this is impossible.  Calculate 24% of 1056 is 253.44px.  Times it by 4 = 1013.76px.  Now take 12px margin space + previous number = 1049.76px.  This works for 1056px width.

Shrink down to 800PX though, and the 12px margins no longer fit within 100%.  You have to use % margins or this does not work.
Comment: Works fine with float: left;  http://jsfiddle.net/KwUcG/2/
Comment: @Michael You've changed the margin to a percentage, I need it to be a fixed margin.
Answer: http://jsfiddle.net/KwUcG/2/
Works fine with float: left;
Another problem you will run into is that you need to use % margins.
I suggest using 1% margins with 25% width columns.  Adding box-sizing: border-box; fixes box sizing issues when floating.
.wrap {
    padding: 0 40px;
    margin: 0 auto;
    background: #e4f;
    overflow: hidden;
}

    .inner {
        max-width: 1056px;
    width: 100%;
        margin: 0 auto;
        background: #34e;
        overflow: hidden;
    box-sizing: border-box;
    }

.fourcol {
    float: left;
    width: 24%;
    margin-left: 1%;
    margin-right: 0;
    background: #ccc;
    box-sizing: border-box;
}

.fourcol:first-child { margin-left: 0; }

Reason why you can't use fixed margin:
Take 1056px width for example.  24% of 1056px = 253.44 times 4 columns = 1013.76 + 3 columns worth of 12px margin space is 1013.76 + 36 = 1049.76.  In this situation, this works with some space left over.
Take an example of 800px wide however...
24% of 800px = 192 times 4 columns = 768 + 36 (3 columns worth of 12px margin space) = 804px
804px is more than the 100% of 800px, and thus it breaks.  You could avoid this to an extent by decreasing 24 to 23 or 22 or 21... But then you are loosing real-estate at large widths.
When using % margins everything adds up...
Comment: This doesn't look responsive... the columns didn't fall into rows. They are simply narrowing width and wrapping the text. You still end up with 4 columns and very skinny ones at that. My search was for "responsive" ie: in a phone the 4 columns convert down to rows.
Answer: unfortunately, you'll have to use percentages for the margin as well. When I shrink the .inner to 300px, 12px is too much, the column itself is 70px. Try using 1% for the margin, and a bit less (for browser incompatibilities) in the width, eg 23.7% instead of 24%
To clarify, the suggested CSS is :
.fourcol {
   display: inline-block;
   width: 23.7%;
   margin-left: 1%;
   background: #ccc;
}

Comment: So is using a fixed margin completely impossible?
Comment: @egr103 If you want to have fluidity then yes.  See the edit in my answer below for an explanation.
Comment: for small resolutions, where a fixed value in the margin becomes a large percentage of the available display, I think so. Eg in the example I gave, the margin is 12px*3 = 36px on a total of 300px space. That's more than 10%, and it is now affecting the page. On a full page ( ie 1056px ), 36px is less than 1% -- completely negligible, so no problems.


Spectral path tracing - image color/brightness incorrect

Question: I implemented a spectral path tracing using physically base BRDF models such as Oren-Nayar,Specular Reflection and Transmission, Lambertian. All calculation in the path tracer uses standard illuminant and macbeth color checker SPD, spectral power distribution. The result of the path tracer for each pixel is the SPD obtained as a sum of SPD obtained from each sample calculated by the pat tracer. This SPD is then converted to CIE XYZ color and to RGB. The result scene obtained is the following one (in this example taking 500 samples per pixel):

As you can see, everything seems fine, except for the brightness/luminance of the scene. Every object in the scene is darker than it have to be. The floor and the front wall of the cornell box in the scene must be white and neutral8 (from macbeth color checker), but they are dark gray.
The following method is the one that trace the samples for a pixel of the path tracer:
Vector3D PathTracer::getPixelColor(const Ray& ray, int bounce) {

    Spectrum<constant::spectrumSamples> L(0.0f);
    int numberOfSamples = 500;
    float sampleWeight = 1.0f/(float)numberOfSamples;

    for (int i = 0; i < numberOfSamples; i++) {

        Spectrum<constant::spectrumSamples> spectrumSample = trace(ray, bounce);
        L = L + spectrumSample * sampleWeight;
    }

    Spectrum<constant::spectrumSamples> Li = scene->light->spectrum;

    ColorMatchingFunction* colorMatchingFunction = new Standard2ObserverColorMatchingFunction();

    //Get tristimulus values.
    Vector3D tristimulus = CIE1931XYZ::tristimulusValues(L, Li, colorMatchingFunction);

    //Convert tristimulus to sRGB.
    Vector3D color = CIE1931XYZ::tristimulusTosRGB(tristimulus);

    //Apply sRGB gamma correction.
    sRGB::sRGBGammaCorrection(color, GammaCompanding);

    //Convert to standard 0 - 255 RGB value.
    sRGB::sRGBStandardRange(color);

    delete colorMatchingFunction;

    return color;
}

As you can seen, I already apply a gamma correction to the color obtained.
Do you have any idea why my image rendered is so dark? My concerns are in the part where I convert the SPD sum, obtained from the samples, into RGB color. Do you see any error? Am I missing something? Do I need other operation to execute a correct conversion from the SPD obtained from the sampling to an RGB color?

To avoid to write a too long question, I will link the main classes used by the path tracer for the calculation:

Lighting/shading class used to recursive calculate the rendering equation https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Shading/PathBRDF.cpp
CIE XYZ conversion https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Color/CIEColorSpaces/CIEXYZ.cpp
sRGB utils https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Color/sRGB.hpp

The other files/classes used are all on this repository (branch luminance)
https://github.com/chicio/Spectrum-Clara-Lux-Tracer/tree/luminance

Thanks you all guys, I hope someone could help me.
Comment: It is perfectly OK if your material looks dark gray if you illuminate it with weak light source. What is the brightness of the light source?
Comment: @ivokabel but my materials should look white and light gray. The SPD of the illuminant used is the D65. Do i need to tweak the spd of the light in some way?
Comment: @ivokabel Do i need to define a brightness paramter and use it somewhere?
Comment: If I am not mistaken, D65 only defines the shape of the spectrum, not the intensity. Therefore, you will really have to add a parameter telling the amount of emitting radiance, or something similar. Related topic is the renderer exposure value, but I saw that you take 1 as the limit value, so you don't have to bother with this one.
Comment: Thank you @ivokabel for the suggestion about the parameter radiance. Could it be just a constant that will be multiplied with the spd of the illuminant during the tracing of rays? Or do i need to multiply the spd of the illuminant during the conversion from spd to cie xyz? Also I don't understand what you mean with renderer exposure value. Where do I take 1 as its value?
Comment: Yes, multiplying the SPD of your illuminant with a value (whether constant or variable) during or before the ray tracing is the way to go.
Comment: @ivokabel what about the render exposure value? Where do i take 1?
Comment: ...and sorry for the confusion about exposure. What I meant is the image value which maps onto maximum value in the resulting picture, 255 in your case. Don't worry about that at this point.
Comment: @ivokabel can you just write a response to my question? In this way your answer an the other (if someone else would response) will remain as reference, and the user will not have to search in the  :) thank you.
Comment: I might get to it tomorrow...
Comment: After re-reading your code, it is unclear to me what is the relation between scene->light->spectrum and material->le in `PathBRDF::shade`. Is it the same value? If yes, do you allow just one light source in your scene? Moreover, why do you normalize your XYZ values with illuminant luminance in `CIE1931XYZ::tristimulusValues()`?
Comment: @ivokabel scene->light->spectrum and material->le contain the same value, the D65 SPD. In the scene init I give to the material of light object the SPD of the scene light. My scenes supports only one light. The normalization in CIE1931XYZ::tristimulusValues() follows the standard conversion from spd to CIE XYZ that you can found here http://www.scratchapixel.com/old/lessons/3d-basic-lessons/lesson-5-colors-and-digital-images/color-spaces/ or on the wiki CIE XYZ page. Is this passage not correct? I don't think so because my engine supports also the whitted ray tracing model that seems to be ok.
Comment: This is an example scene generated with a different illuminant SPD and the Whitted model https://raw.githubusercontent.com/chicio/Spectrum-Clara-Lux-Tracer/master/Screenshots/03_scene4_whittedSpectrum_fl9.png
Comment: In fact if I multiply the material->le with a multiplier the scene become more bright. Here are some rendered images https://drive.google.com/drive/u/1/folders/0BxeVnHLvT8-7Ty1jTVM5U1JJdms. They are not totally correct (as I expect the floor to be white), but maybe i just need a higher multiplier. Do you see any error in code that could avoid this multiplier (some error in the pdf/BRDF calculation)? Thank you very much again @ivokabel.
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/37281/discussion-between-ivokabel-and-fabrizio-duroni).
Answer: The problem lies mainly in CIE1931XYZ::tristimulusValues() function, where you normalize the resulting color to the luminance of your illuminant which causes that directly observed light source has luminance 1, but everything else is much darker. That is a nice thing to do if you just want to visualize colours of various reflectance spectra under a given illumination, but is probably not the best thing to do in a global illumination renderer.
To be more specific, you cannot expect a reflectance profile to be rendered always completely white just because it reflects all incident energy (constant spectral reflectance equal to 1 everywhere). Neglecting the material directional properties (BSDF, etc.), the properties of the light reflected from a point depend both on reflectance spectral profile of the material, and on the spectral and angular properties of the incident light. The incident light therefore depends not only on the SPD of the illuminant used for your light source, but also on its size and distance from the shaded point. The points closer to the light source will be brighter, while distant points will be darker.
I would suggest removing the luminance normalization in your Spectrum-to-XYZ conversion completely and then adjusting the overall brightness of your scene by multiplying the normalized SPD of the used illuminant (D65 in your case) with some brightness parameter. Whether you multiply it before rendering, during rendering or during the XYZ conversion step is completely up to you. The choice shouldn't affect the result.
Comment: Did this solve the problem?
Comment: Hi @ivokabel, sorry for the late reply. You advice was good, and point me in the right direction, but was not enough. Indeed adding a brightness parameter to the light to manage the spectrum of the light. But i also added another parameter to manage the total luminance of a pixel: the tristimulus value obtained from the path tracer is multiply with the total radiance of the pixel (sum of each SPD wavelenght of the pixel). In this way the image brightness is good.
Comment: This is an example: https://drive.google.com/open?id=0BxeVnHLvT8-7MFNuN1pDcG1DZjg. Anyway thank you again. I hope to find a completely physically based solution before the end of my thesis time.
Comment: @FabrizioDuroni, does it mean, that adjusting the power of your light source with the brightness parameter doesn't make the resulting picture bright enough? I think I am missing something and I'm afraid you are adding to much alchemy into it ;-) Anyway, I'm glad it helped at least partially.
Comment: I was missing something too, and in fact i remove that strange "alchemy" that i though was the right thing to do (web is not always a good source :D) Just a brightness multipler is good. This parameter could be substituted with more physically correct values, like the light power (using a Rienmann sum to calculate the integrate the light SPD ) and its area. But for my thesis has been decided that the brightness parameter is enough.
Comment: This is an example of a final rendered image that will be included in the thesis https://drive.google.com/open?id=0BxeVnHLvT8-7TVBZZE44VTRuT1k Keep watching my github repo for the final result ;)
Comment: @fabrizio, but that is almost exactly what I suggested you to do. ;-) And yes, having a more physically meaningful parameter (e.g. power) in a physically-based renderer would be definitely a nicer thing than some unitless brightness parameter. Especially when you have a spectral renderer (it is not clear how to do it in a colour-space renderer). The image looks pretty good to me. Good luck with the thesis!


React - Pseudo element on Input not rendered at all

Question: I am beating my head here over why an ::after element is not rendered at all when being on input element, but on some div it renders just fine.
I am trying to make a performance friendly focus effect on an input element, so when it's in focus, it gets a nice glow from the after element.
Here's what I have
return (

    <input placeholder="Username"></input>

)

and the css
input,
.divider {
  width: 300px;
  height: 40px;
  position: relative;
}

input::after,
.divider::after {
  content: "";
  position: absolute;
  left: 0;
  width: 100%;
  height: 100%;
  box-shadow: 0px 0px 16px 3px #61DAFB;
}

and here's the final product

When inspecting the ::after element for the div, it is right there in the screen, event if I remove its dimensions, but for the input element, it's in the DOM, but when you select it it's nowhere in sight.
Why is that?
Comment: @pwolaq I was only searching for React related stuff, because I thought the problem was specific for react.
Comment: Also it's kind of confusing what is a container and what isnt considering how React forces you to write every html element with a closing tag
Comment: this is not 100% right, you can use the short syntax. Like XHTML does. You don't need closing tag, but you need to close it, like XHTML does.
Answer: :before and :after render inside a container and  can not contain other elements.
Please view specs: https://www.w3.org/TR/CSS2/generate.html#before-after-content


How can you pass multiple primitive parameters to AsyncTask?

Question: There are related questions, such as How can I pass in 2 parameters to a AsyncTask class? , but I ran into the difficulty of trying in vain to pass multiple primitives as parameters to an AsyncTask, so I want to share what I discovered. This subtlety is not captured in the existing questions and answers, so I want to help out anyone who runs into the same problem as I did and save them the pain.
The question is this: I have multiple primitive parameters (e.g. two longs) that I want to pass to an AsyncTask to be executed in the background--how can it be done? (My answer...after struggling with this for awhile...can be found below.)
Answer: Just wrap your primitives in a simple container and pass that as a parameter to AsyncTask, like this:
private static class MyTaskParams {
    int foo;
    long bar;
    double arple;

    MyTaskParams(int foo, long bar, double arple) {
        this.foo = foo;
        this.bar = bar;
        this.arple = arple;
    }
}

private class MyTask extends AsyncTask<MyTaskParams, Void, Void> {
    @Override
    protected void doInBackground(MyTaskParams... params) {
        int foo = params[0].foo;
        long bar = params[0].bar;
        double arple = params[0].arple;
        ...
    }
}

Call it like this:
MyTaskParams params = new MyTaskParams(foo, bar, arple);
MyTask myTask = new MyTask();
myTask.execute(params);

Comment: This strikes me as a lot nicer actually. I've been using the Object... params way of doing it and for some reason it just doesn't feel good or safe to do.
Comment: @DavidWasser : thanx for your update, besides the given solution works great!
Comment: This works, too. But I believe the above options are simpler.
Comment: Shouldn't the class MyTaskParams be public, or at least less restrictive in order to be able to call it from outside?
Comment: @Mouss that depends. In this case, it is a private inner class used only by the surrounding `Activity`. Of course, if you want to put it in a separate file then you would need to make it public or package private.
Answer: Another way: You just need add MyTask constructor in your MyTask class:
private class MyTask extends AsyncTask<String, Void, Void> {
    int foo;
    long bar;
    double arple;

    MyTask(int foo, long bar, double arple) {
         // list all the parameters like in normal class define
        this.foo = foo;
        this.bar = bar;
        this.arple = arple;
    }
    ......   // Here is doInBackground etc. as you did before
}

Then call
new MyTask(int foo, long bar, double arple).execute();

A second way like David Wasser's Answer.
Comment: actually this is my favourite way of the three to pass arguments of different types. No Object to cast and no need to create an additional class.
Comment: You need to call super() in your overridden constructor!
Comment: @zyamys why is super() necessary here? As I understand it, it will be called automatically. See here http://stackoverflow.com/a/2054040/984263
Comment: Out of the options on this thread I like this the best. No helper classes, this class can be used on its own and describes its own requirements. No mystery Objects being passed around. Thanks.
Comment: This works. Feels very dirty but I'm too lazy to try anything else. Can't believe there isn't a nice native way to do this natively. Could possibly support later in Java `<(String,Object,int),Void,Void>`
Answer: It is (strictly-speaking) NOT possible to pass multiple primitives to AsyncTask. For example, if you want to perform myTask.execute(long1, long2) and try to set up private class myTask extends AsyncTask<long, Void, Void> with the corresponding method:
@Override
protected LocationItemizedOverlay doInBackground(long... params) {...}

your IDE will likely complain about needing to override a supertype method. Note that you  are using the so-called Varargs method signature for doInBackground, where (long... params) is like saying "I accept a variable number of longs, stored as an array called params. I don't completely understand what causes a compiler/IDE complaint to be raised, but I think it has to do with how the generic class Params is defined.
In any case, it is possible to achieve what you want with no problem, provided you correctly cast your primitives to their respective non-primitive wrappers (e.g. int => Integer, long => Long, etc.). Actually, you don't need to explicitly cast your primitives to non-primitives. Java seems to handle that for you. You just need to set up your ASyncTask as follows (for the example of longs):
private class MyTask extends AsyncTask<Long, Void, Void> {

    @Override
    protected void doInBackground(Long... params) {
        // Do stuff with params, for example:
        long myFirstParam = params[0]
    }
    ...
}

You can then use this class as you originally intended, e.g.:
MyTask myTask = new MyTask();
myTask.execute(long1, long2);

Or for any number of primitives that you would like, PROVIDED THEY ARE OF THE SAME TYPE. If you need to pass multiple types of primitives, this can also be done, but you will need to modify the above to:
private class MyTask extends AsyncTask<Object, Void, Void> {

    @Override
    protected void doInBackground(Object... params) {
        // Do stuff with params, for example:
        long myLongParam = (Long) params[0];
        int myIntParam = (Integer) params[1];

    }
    ...
}

This is more flexible, but it requires explicitly casting the parameters to their respective types. If this flexibility is not needed (i.e. a single data type), I recommend sticking to the first option, as it's slightly more readable.
Comment: You can also use the following for the method

`protected LocationItemizedOverlay doInBackground(Object[] objects)`

and add the following for the async task definition.

`private class MyTask extends AsyncTask `
Answer: The built in execute method accepts an array of Params, but they all must be of the defined type.. so if you simply set the PARAM type to OBJECT, then you can pass in whatever you like as long as they are children of objects....
private class MyTask extends AsyncTask<Object, Void, Void> {

Then in your doInBackGround, you simply cast each param in order back to what you need it to be:
 @Override
 protected void doInBackground(Object... params) {
     Context t = (Context)params[0];
     String a = (String) params[1];
     List<LatLng> list = (List<LatLng>)params[2];
     .
     .
     .

And your execute is simply:
 new MyTask().execute(context,somestring,list_of_points);

Not as good form as wrapping it in your own wrapper class, or a bundle, or hash or something, because you are order dependent on both sides, but it will work.  Of course you could just make your array a param of HashMap(,) and you basically are custom implementing a bundle at that point, but it will work.
Comment: Folks here are so cool with so many cool technics and this was my favourite!
Answer: I like malajisi's method, but if you didn't, couldn't you use the Bundle class?
 Bundle myBundle = new Bundle();
 myBundle.putInt("foo", foo);
 myBundle.putLong("bar", bar);
 myBundle.putDouble("arple", arple);

Then just pass the bundle and unpack it inside MyTask.  Is this a terrible idea?  You avoid creating a custom class, and it's flexible if you decide you need to pass additional parameters later.
Update: It has been quite a few years since I wrote this answer, and I really dislike it now.  I would recommend against using a Bundle.  If you need to pass multiple parameters into an asynctask (or anything, really), use a custom class that holds all your parameters at once.  Using a bundle is a fine solution to a problem you shouldn't have.  There is no law against creating a custom class to hold exactly what you need, and nothing else.
Also, why aren't you using coroutines?  Asynctasks are so 2014.
Comment: Great idea.  I assume however that this doesn't support custom objects from being passed into doInBackground()
Answer: This is solved via subclassing.
Google has an example for solving this problem (subclassing) in the official Android AsyncTask Documentation:
http://developer.android.com/reference/android/os/AsyncTask.html
Example:
private class DownloadFilesTask extends AsyncTask<URL, Integer, Long> {
    protected Long doInBackground(URL... urls) {
        int count = urls.length;
        long totalSize = 0;
        for (int i = 0; i < count; i++) {
            totalSize += Downloader.downloadFile(urls[i]);
            publishProgress((int) ((i / (float) count) * 100));
                 // Escape early if cancel() is called
            if (isCancelled()) break;
        }
        return totalSize;
    }

    protected void onProgressUpdate(Integer... progress) {
        setProgressPercent(progress[0]);
    }

    protected void onPostExecute(Long result) {
        showDialog("Downloaded " + result + " bytes");
    }
}


Puzzled question of remainder of cubic polynomial

Question: Let $f(x)$ be a cubic polynomial.
If $f(x)$ is devided by $x+2$, the remainder is $-10$.
If f(x) is divide by (x-1), the remainder is 20.
(a) If $f(x)$ is divided by $x^2+x-2$, find the remainder.
(b) If $f(x)$ is divided by $2-x-x^2$, will the remainder the same as the answer in (a)?
I do not know how to solve the question without the polynomial formula, so there I am asking for help.
Comment: Are you sure this is the question? It seems to me we don't have enough information. We see that $x^2+x-2 = (x-1)(x+2)$ but we only know the remainder if we divide by $x+2$ and nothing about $x-1$.
Comment: you 're right so.
Answer: For (a), you have to solve the system of congruences
$$\tag{sys}
\begin{cases}
f \equiv -10 \pmod{x + 2}\\
f \equiv 20 \pmod{x - 1}.
\end{cases}
$$
This is because a solution of this will be of the form
$$
f = f_{0} \pmod{(x + 2) (x -1)},
$$
where $f_{0}$ is a particular solution of (sys), and $(x+2)(x-1)$ is the lcm of $x+2$ and $x-1$, as their gcd is $1$.
There is a standard way for finding $f_{0}$. Find a linear combination of $x + 2$ and $x - 1$ that yields their gcd, which is $1$, or a nonzero scalar multiple thereof: we choose $3$
So you may take
$$ (x + 2) - (x - 1) = 3.$$
Multiply by $10$ to get
$$
10 (x + 2) - 10 (x - 1) = 30 = 20 - (-10),
$$
and thus
$$\tag{sol}
10 (x + 2) - 10 = 10 (x -1) + 20 = 10 x + 10,
$$
and this will be the requested remainder $f_{0}$. Just check using (sol) that any polynomial of the form
$$
(x^{2} + x - 2) g + 10 x + 10
$$
for any $g$, will have the correct remainders modulo $x+2$ and $x-1$.
The fact that $f$ is cubic appears to be irrelevant. Also, the answer to (b) is yes.
Answer: Note that polynomial division of $p(x)$ by $d(x)$ gives $$p(x)=d(x)q(x)+r(x)$$ where $q(x)$ is a polynomial and $r(x)$ has degree less than the degree of $d(x)$. This requires that the polynomials be defined over a field - rationals, reals or complex numbers will do. In such cases $r(x)$ is unique.
We can use this with $d(x)=x+2$ to give $p(x)=(x+2)q(x)+A$, (because $r(x)$ has to be constant) and $p(-2)=-10$. Similarly $p(1)=20$.

Now to the problem itself. Here we do not know $q(x)$ or $r(x)$, but we can write $$p(x)=(x^2+x-2)q(x)+ax+b$$
(the degree of the remainder is at most $1$). If we set, respectively $x=-2, x=1$ we find $$p(-2)=-10=-2a+b$$ and $$p(1)=20=a+b$$
And from this we see $a=b=10$.
It is also easy to check what happens when you change the sign of $r$.


Google App Script - Error when Mapping - TypeError: Cannot read property 'map' of undefined

Question: With the guidance of the community I've recently learned here about BatchUpdating background colors of a given google sheet.
I've attempted to apply this to my actual sheet but run into the error code. TypeError: Cannot read property 'map' of undefined
The code in my test sheet which works without issue is here:
var TestArray = Sheets.Spreadsheets.get("1eAq-RbtrCSMRPZ0p7XIpG3vd29yL-3SQ3D3JGyiUhKg", {
   ranges:"TestBackgroundSheet!A1:AD39", fields:"sheets(data(rowData(values(effectiveFormat.backgroundColor))))"
 });

 var backgroundColors = TestArray["sheets"][0]["data"][0]["rowData"]
                      .map(row => row["values"]
                      .map(value => value["effectiveFormat"]["backgroundColor"]));

I then copied and pasted this code into my actual project. In an attempt to troubleshoot I even copied the exact Sheet from my test project into the actual project. The code is listed here:
var TestArray = Sheets.Spreadsheets.get("1pcIKNUFmkk0d-UGg1sXl5xbsJC2WhocIHpM3et-CMgo", {
   ranges:"TestBackgroundSheet!A1:AD39", fields:"sheets(data(rowData(values(effectiveFormat.backgroundColor))))"
 });

 var backgroundColors = TestArray["sheets"][0]["data"][0]["rowData"]
                      .map(row => row["values"]
                      .map(value => value["effectiveFormat"]["backgroundColor"]));

As you can see the code is exactly the same except for a different SheetID. Despite this i continue to get the TypeError: Cannot read property 'map' of undefined Error.
The error code references line 172 which is .map(value => value["effectiveFormat"]["backgroundColor"]));
Edit:

Edit #2:
With inclusion of Rafa's Code I get this error code -> SyntaxError: Unexpected token , in JSON at position 1 (line 177, file "macros")
Included code is below.
var TestArray = Sheets.Spreadsheets.get("1pcIKNUFmkk0d-UGg1sXl5xbsJC2WhocIHpM3et-CMgo", {
      ranges:"TestBackgroundSheet!A1:AD39",
      fields:"sheets(data(rowData(values(effectiveFormat.backgroundColor))))"
    });

    var rowData = TestArray["sheets"][0]["data"][0]["rowData"]
    .map(row => row.getValues()).toString()

    var backgroundColors = JSON.parse("[" + rowData + "]")
    .map(value => {
         let v = value["effectiveFormat"]
         return v ? v["backgroundColor"] : null
         })

Edit #3:
Logger of RowData with below code
for (var x = 0; x < 40; x++) {
      Logger.log(x + JSON.stringify(TestArray["sheets"][0]["data"][0]["rowData"][x]));
    }

I can see the issue is in the first four lines of Logger statements. While the empty rows in the "Testing Sheet" returns full rows of White background color (RGB {"green":1,"red":1,"blue":1}) In the "Reat Sheet" I get an empty object for 3 of the first 4 lines.  I've include the first four lines out output below. I'm not sure why this occurs.
Actual:
[20-10-21 08:53:23:591 EDT] 0{}
[20-10-21 08:53:23:593 EDT] 1{"values":[{},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}}]}
[20-10-21 08:53:23:595 EDT] 2{}
[20-10-21 08:53:23:597 EDT] 3{}
[20-10-21 08:53:23:600 EDT] 4{"values":[{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}}]}

Testing:
[20-10-21 05:53:14:167 PDT] 0{"values":[{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}}]}
[20-10-21 05:53:14:170 PDT] 1{"values":[{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}}]}
[20-10-21 05:53:14:193 PDT] 2{"values":[{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}}]}
[20-10-21 05:53:14:195 PDT] 3{"values":[{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}}]}
[20-10-21 05:53:14:198 PDT] 4{"values":[{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"blue":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"blue":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"red":1,"green":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"green":1,"red":1,"blue":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"red":1,"green":1}}},{"effectiveFormat":{"backgroundColor":{"blue":1,"green":1,"red":1}}}]}

Comment: Try the api explorer or Log `console.log(JSON.stringify(TestArray["sheets"][0]["data"][0]))`
Comment: `TestArray["sheets"][0]["data"][0]` doesn't have `rowData`
Comment: I'm sorry I don't fully understand. How would it not have rowData? I'm running the function on an exact copy of a sheet that's in a different spreadsheet. The "Test" version much have rowData because it work. The "Actual" version doesn't? Is there a command to prove whats in that Array?
Comment: When the sheet has no values and the default background color of all cells, your request returns `{"sheets":[{"data":[{}]}]}`. In this case, `rowData` is not included. I think that by this, such error occurs. So in order to correctly understand about your current situation of `In an attempt to troubleshoot I even copied the exact Sheet from my test project into the actual project.`, can you provide the image of the sheet `TestBackgroundSheet!A1:AD39` in Spreadsheet `1pcIKNUFmkk0d-UGg1sXl5xbsJC2WhocIHpM3et-CMgo`?
Comment: @Tanaike - Thank you for the reply. I've edited my original question and included the requested screenshot. It is edited minimally because everything that is and "F" were names that I felt uncomfortable posting. Please let me know if there is anything else I can provide.
Comment: Thank you for replying and adding the information. When the sheet of `TestBackgroundSheet!A1:AD39` on the Spreadsheet of `1pcIKNUFmkk0d-UGg1sXl5xbsJC2WhocIHpM3et-CMgo` is the additional image, `rowData` is included in the returned values and no error occurs at the script. So unfortunately, I cannot replicate your situation. This is due to my poor skill. I deeply apologize for this. When I could correctly replicate your situation, I would like to think of the solution.
Answer: Answer:
Not all elements of row["values"] have data - and so you can't run .map(value => value["effectiveFormat"]["backgroundColor"]) when that row doesn't have data.
More Information:
You've got your data from the API using
Sheets.Spreadsheets.get("1eAq-RbtrCSMRPZ0p7XIpG3vd29yL-3SQ3D3JGyiUhKg")

and filtered it accordingly with your field mask. The data you get as a response will be everything in the sheet - even the cells that do not have data for background colour. As a result of this, you can not map each row like this, as you will try and reference the effectiveFormat element where it simply doesn't exist.
Fix:
You can use the ternary operator to solve this; if the element value["effectiveFormat"] doesn't exist you can simply return null:
var rowData = TestArray["sheets"][0]["data"][0]["rowData"]
    .map(row => row.getValues()).toString()

var backgroundColors = JSON.parse("[" + rowData + "]")
    .map(value => {
        let v = value["effectiveFormat"]
        return v ? v["backgroundColor"] : null
    })

NB: The API also returns functions within the JSON objects in the response which you can use in Apps Script. This comes in handy because referencing row["values"] directly can return the object rather than the data:
console.log(TestArray["sheets"][0]["data"][0]["rowData"]
            .map(row => row["values"]))

yields:
[
  {
    setPivotTable: [Function],
    getDataSourceTable: [Function],
    getDataValidation: [Function],
    getEffectiveValue: [Function],
    setNote: [Function],
    setFormattedValue: [Function],
    getTextFormatRuns: [Function],
    setUserEnteredFormat: [Function],
    toString: [Function],
    getFormattedValue: [Function],
    setEffectiveFormat: [Function],
    effectiveFormat: [Object],
    setDataSourceFormula: [Function],
    getPivotTable: [Function],
    setUserEnteredValue: [Function],
    setDataValidation: [Function],
    setDataSourceTable: [Function],
    getUserEnteredFormat: [Function],
    setEffectiveValue: [Function],
    getEffectiveFormat: [Function],
    getHyperlink: [Function],
    getNote: [Function],
    setHyperlink: [Function],
    getUserEnteredValue: [Function],
    setTextFormatRuns: [Function],
    getDataSourceFormula: [Function]
  },
  ...
]

The toString() call after the first .map(row => row.getValues()) gets past this before putting the data back into the second mapping function.
References:

Conditional (ternary) operator - JavaScript | MDN
Comment: thank you for the response. I will do a lot of testing and reading about your response today. In my first read of your response I didn't see why this would work in my test instance but not the real one despite the spreadsheet being the same?
Comment: Without seeing both spreadsheets it's impossible to say, though suffice to say they can't be identical if the code works on one but not the other. Likely some hidden setting or structure diff
Comment: Hmmm, ok. I'm not sure where to start looking for that difference. I've literally used google sheets built in "Copy To" -> "Existing Spreadsheet" Function. It should be identical...
Comment: I've updated my original Question to include an Edit that incorporates your suggested code in your Answer. Doing this i get the error -> SyntaxError: Unexpected token , in JSON at position 1. This occurs at line -> var backgroundColors = JSON.parse("[" + rowData + "]")
Comment: Can you log `rowData` and paste its structural output here?
Comment: I believe I've done as you asked and added it as Edit #3 in the original Question.
Comment: I don't see an `Edit #3`.
Comment: I apologize. I attempted to post both these items at the same time and then was called away while posting the second edit. It's posted now and I've shown the difference between the two. Not sure why this difference is occurring. Thanks again for the guidance.
Comment: Glad to help! It looks like however you're defining `rowData` means that it's picking up different data for each sheet. Without looking at them it is impossible to say why, but also this is beyond the scope of your orignial question. I think that it would be best to pose this as a new question dedicated to the issue as it's beyond what this question was originally asking (map being undefined was because of empty rows being picked up, not *why* there are empty rows)
Comment: Your point about starting a new question regarding "why" is more than fair. Your initial answer suggested a fix for dealing with the empty rows and null value using the ternary operator. When implementing it I'm still getting the error code `SyntaxError: Unexpected token , in JSON at position 1` as described in edit #2 of the original question. Any suggestions there?
Comment: Edit #2, `SyntaxError: Unexpected token , in JSON at position 1` is a different error to the original question `TypeError: Cannot read property 'map' of undefined` and came about after the original error was rectified. You error is suggesting that `rowData` starts with a `,`, hence the string `"[" + rowData + "]"` has a `,` at position 1.


Xcode 4 warning "Expression result unused" for NSURLConnection

Question: I'm just trying to do my usual data transfert. I define my NSMutableURLRequest then call
[[NSURLConnection alloc] initWithRequest:request delegate:self];
This used to be ok with Xcode 3 but Xcode 4 warns me about "Expression result unused" on that line.
The request does work but I would like to find a way to get rid of the warning.
I suppose I could store the connection in a variable but I don't really need it and I can't see the point of setting it to nil the next line (although this would remove the warning)
Please note: I'm not 100% sure if it's Xcode 4 or the fact ARC is enabled.
Answer: When a function returns a result that you don't need you can cast it to void to eliminate the compiler warning:
(void) [[NSURLConnection alloc] initWithRequest:request delegate:self];

I haven't used ARC yet so I can't say if this is a good idea, before ARC you would need to keep this pointer result somewhere so you could release it.
Comment: Yes, because it documents the fact that you know there is a result being returned but you don't need that result.
Comment: This did remove the warning and the function is still working as required so all good! It's the first time I see this cast to void. Is it good practice? Thanks!
Comment: I still get the warning when I am attempting to use a #define based logging macro:  `#if SHOW_DEBUG_LOG
# define DEBUG_LOG(x,...)  NSLog(@"Debug Log: "x, __VA_ARGS__)
#else
# define DEBUG_LOG(x,...)  (void)(x); do { (void)(__VA_ARGS__); } while (0)
#endif`
Answer: progrmr's answer is correct, but here's an even cleaner way to do it:
[NSURLConnection connectionWithRequest:request delegate:self];
This doesn't cause a warning, even if you don't cast the result to void.
Comment: Deprecated as of iOS 10. :(
Answer: Someone should be responsible for that NSURLConnection. It is not needed to store the connection but it is better coding if you do. The problem is that after you created our NSURLConnection no one has a pointer to that created instance which should not be the case.
Let's assume the following example:

your instance of ClassA is creating an instane of NSURLConnection
your instance of ClassA is beeing released and dealloced
NSURLConnection is still alive and will fire the delegate to your deallocated instance.

To solve that problem you should store the instance of NSURLConnection and should release that connection if your instance of ClassA is being dealloced which results in deallocating the instance of NSURLConnection as well.
Comment: but the problem still exist semantically: `NSURLConnection` knows your instance of `ClassA` but nobody knows your instance. There is a relationship between both instances and this relationship needs to be managed ;) But if you dont save the pointer to `NSURLConnection` then nobody has a pointer to it and either ARC is deallocating the connection immediately or never. If you save the resulting pointer then the warning will not appear and you have a better coding-design and you are then able write more reliable code:`connection:(NSURLConnection*)con didSth{if(con != self.curCon) return;}`
Comment: I can't release it as ARC is on. Is it not what ARC is supposed to do automatically, deallocating the pointer?

Xcode 4 warning "Expression result unused” for NSURLConnection


Data scraping with scrapy

Question: i want to make a new betting tool, but i need a database of odds and results and can't find anything in the web. I found this site that has great archive: OddsPortal
All i want to do is scrape the results and the odds from page like the one above. I found that a tool called Scrapy can do it, is it true? Can someone help me with some hints?
Comment: Possible copyright issues aside, you may have trouble getting useful answers here.  This site focuses on specific programming questions: "I tried `this`, and got `that` as a result".   It doesn't work tend to well with open-ended questions like yours.
Comment: ty for the answer, and where i can ask for this?
Comment: In this case, it appears that Austin gave you a `very` good answer. Please remember to `Accept` it. In the general case, I'm not sure - I'd have probably just downloaded Scrapy and started trying, myself. If I ran into trouble, I'd have then come here or the Scrapy forum.
Answer: I don't know about Scrapy, but JSoup should help you get you started.
http://jsoup.org/
Download the .jar file. Right click your project folder > Properties > Java build path > libraries > add external jars > find the jar and click it.
It's a nice little HTML parser
Here's an example.
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class HtmlParser {

public static void main(String[] args) throws IOException{
    String url = "http://stackoverflow.com/questions/16794913/data-scraping-with-scrapy";
    Document document = Jsoup.connect(url).get();

    String question = document.select(".question .postcell").text();

    System.out.println(question);
}

This will print your question :P
Right click this webpage and hit inspect element.
Then find the element you want and put the name of it's class (or ID) in the document.select() part in this case ".question .postcell". (.parentClass .childClass)
If you need extra help there is a guide on the jsoup website
Hope this helps!
EDIT:
I was board so I threw together a little something that fetches all the Italy soccer scores in the 2003-2004 year,  enjoy :D - link - http://www.oddsportal.com/soccer/italy/serie-a-2003-2004/results/
public class HtmlParser {

    String url = "http://www.oddsportal.com/soccer/italy/serie-a-2003-2004/results/";
    Document document = Jsoup.connect(url).get();

    Elements stats = document.select("#tournamentTable tbody tr");

    for(int i = 0; i < stats.size(); i++){
        System.out.println(stats.get(i).text());
    }
}

Output:

Soccer» Italy»Serie A 2003/2004

1 X 2 B's

AC Milan - Brescia 4:2 - - - 6

Chievo - Bologna 2:1 - - - 5

Empoli - Inter 2:3 - - - 5

Parma - Udinese 4:3 - - - 5

Lazio - Modena 2:1 - - - 4

Lecce - Reggina 2:1 - - - 5

Perugia - Ancona 1:0 - - - 1

Sampdoria - AS Roma 0:0 - - - 4

Siena - Juventus 1:3 - - - 5

1 X 2 B's

Ancona - Empoli 2:1 - - - 1

AS Roma - Perugia 1:3 - - - 3

Bologna - Lecce 1:1 - - - 7

Brescia - Lazio 2:1 - - - 1

Inter - Parma 1:0 - - - 7

Juventus - Sampdoria 2:0 - - - 7

Modena - Siena 1:3 - - - 7

Reggina - AC Milan 2:1 - - - 1

Udinese - Chievo 1:1 - - - 3

1 X 2 B's

AC Milan - AS Roma 1:0 - - - 6

Parma - Ancona 3:1 - - - 3

Lazio - Reggina 1:1 - - - 6

Lecce - Inter 2:1 - - - 6

Perugia - Juventus 1:0 - - - 4

Sampdoria - Udinese 1:3 - - - 5

Siena - Brescia 0:1 - - - 3

1 X 2 B's

Ancona - Chievo 0:2 - - - 3

AS Roma - Empoli 3:0 - - - 6

Inter - Lazio 0:0 - - - 6

Juventus - Lecce 3:4 - - - 6

Modena - Sampdoria 1:0 - - - 5

Reggina - Parma 1:1 - - - 5

Udinese - AC Milan 0:0 - - - 6

1 X 2 B's

Lazio - AS Roma 1:1 - - - 7

1 X 2 B's

Modena - AS Roma 0:1 - - - 6

Chievo - Reggina 0:0 - - - 4

Empoli - Brescia 1:1 - - - 5

Parma - Juventus 2:2 - - - 6

Inter - Bologna 4:2 - - - 6

Lazio - Ancona 4:2 - - - 5

Sampdoria - Perugia 3:2 - - - 6

1 X 2 B's

Lecce - Udinese 2:1 - - - 6

Siena - AC Milan 1:2 - - - 5

1 X 2 B's

Perugia - Inter 2:3 - - - 7

1 X 2 B's

Juventus - Lazio 1:0 - - - 7

AC Milan - Empoli 1:0 - - - 7

Ancona - Bologna 3:2 - - - 7

AS Roma - Chievo 3:1 - - - 7

Brescia - Modena 0:0 - - - 7

Reggina - Udinese 0:1 - - - 7

Siena - Sampdoria 0:0 - - - 7

so cool!
Comment: thank you very much! i own you a beer! ;)
Comment: @Austin Excellent answer.  Too bad the community decided to close the question.

Data scraping with scrapy


How to manually store image in (database) table in Visual Studio 2012 and retrieve using jQuery Ajax in ASP.NET MVC

Question: My database table
First I tried storing images with datatype image/varbinary but I get an error:

Invalid Value The changed value in this cell was not recognized as being valid. .Net Framework Data Type: Byte[] Error Message: You cannot use the result pane to set this Field data to values other than NULL Type a value appropriate for the data type or press ESC to cancel the change

How can I store image in my database table manually and retrieve them through jQuery Ajax method?
Comment: can you show your upload code / action?
Answer: On server, you convert image to a base64 string and save to database.
using (Image image = Image.FromFile(Path))
{
    using (MemoryStream m = new MemoryStream())
    {
        image.Save(m, image.RawFormat);
        byte[] imageBytes = m.ToArray();

        // Convert byte[] to Base64 String
        string base64String = Convert.ToBase64String(imageBytes);
        return base64String;
    }
}

Beside Client, you put base64 string in Image object
var image = new Image();
image.src = 'data:image/png;base64,your base64 string';
document.body.appendChild(image);

I think store image on the disk better.
Answer: I'm not sure if this works in VS2012, but you can try this approach that I'm using in my MVC projects:
public action UploadFile(HttpPostedFileBase file) //this is how you get the file that was submited from your form
{
    //so now that I have the file, i need to save it somewhere
    //let's see if we have a folder for that, if not let's create it

    if(!Directory.Exists(Server.MapPath("~/The path from the root of the app to the uploaded files folder/")))
    {
        Directory.CreateDirectory(Server.MapPath("~/The path from the root of the app to the uploaded files folder/"));
    } //the ~ symbol is part of the code, it returns the physical path to the root of the application
    var storage = Server.MapPath("~/The path from the root of the app to the uploaded files folder/");

    //now we need a name, we can generate one as a GUID for example, or we can get it's own.

    var fileName = file.FileName.Substring(file.FileName.LastIndexOf("\"), file.Name.Length + 1);

    //Now you might have to play a bit with the substring arguments, I'm not sure i wrote that right,
    //it might not need the +1 part, or it might need it on the LastIndexOf as well,
    //anyway, now we have the name and the folder where to save our file, let's do that:

    path = String.Format("{0}\{1}", storage, filename);
    try
    {
        file.SaveAs(path); //Now the file is on the server,
        //just save this path and whatever else you need in the database so you can
        //grab it later.
    }
    catch
    {
        //do your exception handling here
    }

    Saving to database here...
}

In order to grab it you just get that object from the database like any other objects like so:
Let's presume you're doing everything in your controller, which you shouldn't,
public Action GetFile(int id)
{
    using(context = new YourDbContext())
    {
        var file = context.Files.FirstOrDefault(p => p.Id == id);
    }
    return View(file);
}

And put it's path for the src in html to show the image like so:
 //Considering that in your controller you will `return View(file);`
//So your file object is the model.

One more thing, I'm not sure if in paths you'll need \ or /, I always get it wrong the first time so debug it on the first run and if your path looks likeFolder1Folder2Folder3File it means you need to change it :).
I hope this helped, good luck.
P.S., there might be better ways to do it, but this is how I'm doing it and it seams to work fine for me.
Answer: Have you ever tried other ways to store images ?
I have never used such a way to store image in database if you want my two cents it would be better to save your image on host and save its url in the database.
You only need url to show picture in browser...!
Answer: use in C# Button Submit with data source
   if (FileUpload1.PostedFile != null && FileUpload1.PostedFile.FileName != null)
            {
            byte[] img = new byte[Request.Files[0].ContentLength];
            Request.Files[0].InputStream.Read(img, 0, Request.Files[0].ContentLength);
            datasource.Photo = img; --here you can save file to Database
            }

for retrieving image from dbserver use belowed
img.ImageUrl = "data:image/jpg;base64," + Convert.ToBase64String(Upprofile.Photo);


Custom Sorting in Linq to Entities based on QueryString

Question: Spent all day trying to find ready to use solution for "Sort data in LINQ bases on Query String" without any results.
SO, I have a LINQ query in action:
public AcrionResult MyAction(int perPage = 10, string orderBy = "DESC", sting sortBy = "id")
{
    var some = from m in db.Some select new ExampleModel {id = m.id, some = m.some};
    return View(some);
}

From exmaple above:

1. perPage describe how many items we should show in page
2. orderBy DESC or ASC order
3. sortBy can be some field from ViewModel
I need to make somethisng like this:
var query = some.OrderFilter(...).AmountFilter(...).SortByFilter(...)
Can somebody help me?
Thanks a lot!
Comment: https://stackoverflow.com/questions/2013198/linq-to-sql-dynamic-orderby-case-when/56868434#56868434
Answer: You could use dynamic LINQ. Here's another example of implementing dynamic queries.
Answer: The page size is best handled by Take and Skip – LINQ to Entities will translate that in to SQL.
The parametrised sort can be done either by creating an expression tree based on the selected property and passing it to OrderBy or OrderByDescending as needed. Or by using Entity-LINQ building the SQL from the name of the field.
The latter apporoach needs to white-list the column names to avoid SQL Injection..


How to make a worldborder that expands every in-game day at dawn?

Question: I want to make a world with a world border that increases 2 blocks every in game day when the sun rises using command blocks. How would I accomplish this?
I don't completely understand commands so please be really detailed in how to do this.
Comment: @MrLemon  dont think it is a duplicate because that tells you how to make  it expand with time and will not take into sleeping i just want it to expand every time the sun rises
Comment: possible duplicate of [How to get a slowly expanding world border in Minecraft?](http://gaming.stackexchange.com/questions/181937/how-to-get-a-slowly-expanding-world-border-in-minecraft)
Answer: Both of these solutions require you to have set up your worldborder first, and to know the location of your spawn chunks (the chunks around your world spawn). They need to be built in the spawn chunks so that they are always loaded.
Simple solution
Place a daylight sensor on top of a command block with the following command:
/worldborder add 2

Make sure the daylight sensor is exposed to daylight. The major downside here is that it is not exactly hidden.
Hidden solution
For a server or just general asthetics, it might be good to hide the command blocks deep underground at bedrock level. Unfortunately, this is a little more complicated, but totally doable.
Create a scoreboard objective called daytime using
/scoreboard objectives add daytime dummy

As pointed out by Skylinerw on the Minecraft forums, daytime does not actually reset to 0 every day. Sleeping in a bed advances the daytime forward to a multiple of 24000 instead of setting it to 0. To make this work, we create a second objective and use it to store a number (24000) in a fake player called "#DAYLIGHT" (The "#" is illegal in player names, making this unique).
/scoreboard objectives add MATH dummy
/scoreboard players set #DAYLIGHT MATH 24000

Now, create a setblock or fill-clock running the following commands.
/time query daytime
/stats block <x> <y> <z> set QueryResult @a daytime
/scoreboard players operation @a daytime %= #DAYLIGHT MATH
/execute @p[score_daytime=1000,score_daytime_min=1000] ~ ~ ~ worldborder add 2

where <x> <y> <z> are the coordinates of the command block with the first command (relative coordinates using ~ work).
Comment: i cant get your hidden solution to work i dont know what im doing wrong. So far I have a setclock with 3 command blocks around the redstone block with your 3 commands in them and when i do /time set 1000 nothing happens
Comment: @RNH good comment, I just saw that I actually forgot something. You have to replace `  ` with the coordinates for the first command block.
Comment: I did that and still the world border did not move
Comment: @RNH Hm... it worked for me when I tested it. Then again, I used `time set 990` to test it. Try that, and if it works, set the execution time to 1010 instead. Also, use `scoreboard objectives setdisplay sidebar daytime` to check if the scoreboard updates correctly
Comment: On the third command block i get this message underneath the command "The entity UUID provided is in an invalid format" maybe that has something to with it not working
Comment: @RNH That's the error you get when the target selector doesn't resolve a valid target. NULL is not a valid UUID.
Comment: alright so when i add the sidebar it says my name and then 0 so it must not be updating or something but in the chat it says "storing Querystat results in daytime on @a"
Comment: "Unfortunately, this is a little more complicated, but totally doable." - couldn't you simply hook up the daylight sensor to an inverter, the command block on the inverter output, and the whole thing encased in bedrock? Covered daylight sensors are inverted IIRC.
Comment: @Random832 Yeah, *always*. As in, it's not reacting to dawn in any way, because it can't see the dawn.
Comment: @KRyan That's not what the wiki says.
Comment: @Random832 Ah, I misread your comment. The wiki, at least, confirms your idea.
Comment: Its ok i got a fix for my idea http://www.minecraftforum.net/forums/minecraft-discussion/redstone-discussion-and/command-blocks/2468502-command-block-help
Comment: @RNH great find. I incorporated it into the answer.
Answer: First place a daylight sensor with a comparator infront of it, with the comparator facing out of the daylight sensor. Then put a command block as the output of the comparator. The redstone should look like following:

Then, inside the command block, type:
/worldborder add 2

How this works is that the daylight sensor will activate each day and when it activates, it will give an input to the comparator which will then execute the command inside the command block and add a distance of 2 to the world border.
Comment: @RNH Well, you should accept the method that you used and worked for you.
Comment: Duh. You answered faster than me. Edit: I edited my answer and let him know that your take on daylight sensors can be better.
Comment: @ardaozkal Thank you, but it seems he chose you anyways.
Comment: I used your method in the end
Answer: Plan 1: (hard&redstone way, no big differences with Plan 2) (not tested)
First of all, you must do a 1 day loop with redstones. And next off, when you connect them to the command block, you must add /worldborder add 2 1 (2 = 2 blocks, 1 = 1 seconds to open) to it.
Plan 2: (easy way) (tested and works)
Put a Daylight Sensor, connect it to a command block, add /worldborder add 2 1 (2 = 2 blocks, 1 = 1 seconds to open) in command block.
Note: Plan 2 might not work, see CommandFox' answer for a better take of this plan.
Plan 3:  (tested and works) (world will grow slowly, a new block everymidday)
A minecraft day is 20 minutes. 1200 seconds. So just use this code in console: /worldborder add 2xdays-you-want 1200xdays-you-want. So making this work for 1000 days would be /worldborder add 2000 1200000.
Also if you are mapping, I recommend adding another command block to let the player know that world grew.
Comment: will the daylight sensor still work if i have it underground
Comment: @RNH unfortunately and AFAIK no.
Comment: ok so i tested it and the daylight sensor will work underground but it will turn on at night instead of day


Disney Infinity 3.0 cross-platform playsets and characters missing

Question: I downloaded Disney Infinity 3.0 game on Windows and bought the Rise Against the Empire and The Force Awakens digitally, and got my kids a couple physical characters.  For Xmas I bought my kids an Xbox One so they'd let me have my computer to myself, and got the Twilight of the Republic starter pack, assuming it would complete our set.  But after installing the game on the Xbox, it will only let me play the playsets for which we have a playset piece that we can stick on the base, and only the characters with physical toys that we can put there.  I've searched all over the web to investigate this issue and I can't find a single mention of not being able to use "web codes" on the Xbox.  It doesn't make any sense to me.  Why are there "web stores" and online accounts  if you can't use them on all platforms?  Are there any options for me other than double-purchasing the RATE and TFA playsets that I already spent $50 on?  (What else is a bummer is that we spent hours trying to kill the AT-ATs on Hoth before we actually cleared the zone...)
Answer: I believe the console version of the game requires the physical toys, so purchasing the digital versions will not help you play the characters on Xbox.
On the other hand, the physical characters come with a web code, so you can add them to your account.  This allows you to do things in the other direction - buy a character for the console, and then play them in the digital version.


End of Input Error in Tellraw

Question: I have a /tellraw command, straight from my map:
 /tellraw @a [{"text":"I am a "},{"text": "DOOFUS",
 "color":"red"},{"text":".
 "},{"text":"Yes","color":"red","clickEvent":{"action":"run_command","value":"/tellraw
 @a [{\"text\":\"You are \"},{\"text\":\"OFFICIALLY
 \",\"color\":\"red\"},{\"text\":\"a
 \"},{\"text\":\"DOOFUS\",\"color\":\"red\"},{\"text\":\"\nGo to the
 bottom-left of the map
 name.\"}]"}},{"text":"No","color":"green","clickEvent":{"action":"run_command",
 "value":"/tellraw @a {\"text\":\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\"}"}}]

When I run it, the chat reports I am a DOOFUS. YesNo, but when I click Yes, it says Invalid json: End of Input at line at line 1 column 89. Clicking No works.
Comment: Minecraft commands sure are confusing. Answerers: Curly bracelet counts are equal.
Comment: If Yes doesn't work, then there's something wrong with your Yes command.  Try singling that out, and see what you can find.
Answer: When a player runs a command from a click event via /tellraw, they are running the command as if they have typed the command themselves into the chat. This means they are subject to standard chat limitations, such as the requirement of being OP'd and a 100-character limit.
The command you're trying to have them run exceeds that limit at 192 characters.
You will need to use a shorter command instead. If OP status is a concern, you have to use /trigger, which will modify the executor's score of the "trigger" type without needing to be OP'd.
Prerequisites

Adding the "trigger" objective-type, which is the only objective-type that can be used with the /trigger command (meaning you cannot use "dummy").
/scoreboard objectives add CLICKED trigger

The commands you would run in /tellraw's click event would instead be:
/trigger CLICKED set 1
/trigger CLICKED set 2

Clock commands
The following must be run in numerical order on a clock.

If a player activates their trigger, it becomes locked and cannot be activated again. You can constantly enable their trigger to avoid locking, though it would be up to you to determine if there's a need to keep it locked.
/scoreboard players enable @a CLICKED

Run the nested /tellraw command if the player clicked "Yes", which set their score to 1.
/tellraw @a[score_CLICKED_min=1,score_CLICKED=1] [{"text":"You are "},{"text":"OFFICIALLY","color":"red"},{"text":"a"},{"text":"DOOFUS","color":"red"},{"text":"\nGo to the bottom-left of the mapname."}]

Run the nested /tellraw command if the player clicked "No", which set their score to 2.
/tellraw @a[score_CLICKED_min=2,score_CLICKED=2] {"text":"\n\n\n\n\n\n\n\n\n\n"}

Reset the player's "CLICKED" score to 0.
/scoreboard players set @a[score_CLICKED_min=1] CLICKED 0

Answer: Here is another (possibly easier) way:
Make the Yes command run a /setblock x y z redstone_block. The x y z are linked to two command blocks with what the Yes was originally connected to (except without the extra backslashes), and a /setblock x y z air.
Comment: Be aware that this requires the clicking player to be OP'd and does not support multi-player because you don't know who had clicked.
Comment: My map is singleplayer, so it should work. But yes, this only works in singleplayer.

End of Input Error in Tellraw


How to configure an Ubuntu server so that all traffic goes through a VPN?

Question: This question is Not about how to set up VPN on an Ubuntu server.
Edited: I think what I'm looking for is the way to set up a PPTP VPN client on my Ubuntu server by command line.
I have an Ubuntu server running and connected to the internet. I also have a VPN service (PPTP) running on another Linux machine. Now all I want to do is configure this Ubuntu server so that all the internet traffic it generates goes through the VPN.
How can I achieve this?
Answer: Hi You can try this below link
you need to install pptpd
http://kamleshgosavi.blogspot.in/2011/01/vpn-server-on-ubuntu.html
Comment: In my question I made it very clear (the first line) I am not asking how to set up a VPN server on Ubuntu.
Answer: Change the default route on your server to point to the machine which has the PPTP client running.  In /etc/network/interfaces, change the gateway line to point to the PPTP client machine, then restart the machine (or just restart the interface, if you're feeling frisky.  or just change the default route manually with the route command, if you're feeling extra frisky).
You'll also need to make certain that the machine with the PPTP tunnel up is willing to act as a gateway for traffic - this last can be done precisely with iptables, or it can be done more simply with sysctl.  To do it the simple way - enable acting as a gateway for any traffic it knows a route for - uncomment the net.ipv4.forward line in /etc/sysctl.conf, and reread it:
sudo sed -i s/\#net\.ipv4\.ip_forward/net\.ipv4\.ip_forward/ /etc/sysctl.conf
sudo sysctl -p

Hope this helps.


Laravel 7 Sanitizing Form Data

Question: I'm new to Laravel 7 and wondering if there's an out-of-the-box, elegant solution to sanitizing HTML form inputs? Or maybe a trusted third-party package I can download that you recommend? This is for data I will store in a database. Thanks for any help.
Answer: One recommended out-of-the-box way of sanitizing data is using the filter_var function that comes with PHP in conjunction with the different sanitize filters. By the way, this is also a cool way to validate input, take a look at the types of filters to find out more.
When working in Laravel projects, I like to use voku/portable-ascii library, because it's already a framework dependency. It is a nice assortment of functions to clean input, remove non-printable characters, and to generally transform any input into ASCII, complete with transliteration and whatnot. It's not always perfect, but usually good enough and gets the job done.
It always depends on what you want to sanitize, how, and why. In many situations you do not need to sanitize the input at all if you stick to the best practices. When working with Eloquent or the Query Builder, your data is automatically escaped and on retrieval, when you output it e.g. via {{ $data }}, they will be properly escaped too.
There are some situations where you should be more cautious, especially if you are handling the raw user input yourself and probably passing it to the system in command line parameters, filenames or such. In those cases it is usually a good idea to be as restrictive as possible and as permissive as necessary. Sometimes a good old preg_replace('/[^0-9A-Z_-]/i', '', $subject) is just the right choice. If you want to be as permissive as possible, give the suggestions above a try.
Comment: I should've stated this in my question. The data I need to sanitize is quite standard: Name, email address, phone number and a checkbox. That's it! :-)
Comment: The PHP filter_var function you mentioned looks cool, and it does do emails, but U.S. phone numbers are oddly omitted.
Comment: The sanitize filters will take care of the email sanitizing for you. Names don't really need to be sanitized usually, unless you want to strip out annoying icons, RTL characters and such. Phone numbers... well, do you really want sanitizing or validation here? Sanitizing is easy: `preg_replace('/[^0-9\(\)\- ]/', '', $number)` should take care of most of it. You can also use HTML5 form validation in addition, see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/email or tel etc. For validation take a look at https://laravel.com/docs/7.x/validation.
Comment: It is not odd that U.S. phone numbers are omitted. Don't confuse sanitizing with validation. To do plausibility checks on phone numbers, you would need rulesets or even databases that are very specific from country to country. If you need very basic validation, regex is your friend. There are already hundreds of answers on that topic, like https://stackoverflow.com/questions/9776231/regular-expression-to-validate-us-phone-numbers
Answer: I was just wondering if there was some good package to deal with sanitization using Laravel Framework and found this:
https://github.com/elegantweb/sanitizer
Worked like a charm and has support to Laravel 10.
Enjoy!

Laravel 7 Sanitizing Form Data


Where does the Illusion World song from Chocobo Racing come from?

Question: I'm sure I've heard this song before, even though I haven't seen much Final Fantasy gameplay (since I've never owned a console I've never had any of those games).
Where does it come from? (Bonus point for a Youtube link.)
Comment: I didn't feel like creating an 'identify-this-song' tag :)
Comment: Eh, [identify this game seems appropriate](http://gaming.stackexchange.com/questions/7790/what-is-this-sound-from).
Answer: Some googling about the Chocobo Racing OST brought me to the Final Fantasy Wiki, which contains some YouTube videos with a few tracks from the OST.
One of the tracks happens to be Illusion World, which is the one played in the video.
According to the Wiki, the song is from the final boss battle in Final Fantasy III, where it was called 最後の死闘 (Saigo no Shitō), which is best translated as "Battle to the Death".
Answer: I actually answered this roughly 40 minutes ago in Gaming's The Bridge Chat, but Gaming is blocked from where I work... would have been back sooner (stupid traffic jams).
This is the Final Boss Theme from Final Fantasy III for the NES/DS.


Correct User Defined Version of Map Function in Mathematica?

Question: I'm trying to create a user defined version of the Map[] function in Mathematica and I'm running into a few problems.
Here is what I have so far:
map[x_, s_List] := mapAux[x, s, {}];
mapAux[x, s, {}] :=  Append[{}, First[s]];
mapAux[x, Rest[s], {}];

I'm trying to use it as
map[# + 1 &, {3, 6, 8}]

but this gives a mysterious error beside the output:
 Rest::normal: Nonatomic expression expected at position 1 in Rest[s].

 mapAux[#1 + 1 &, {3, 6, 8}, {}]

The ideal result  would be {4,7,9}. I researched the "Nonatomic expression" error and I'm not sure what it means. I'm passing a list to it, but it's just exploding!
Comment: You might find my previous answer (and the comments on it) to the more-or-less exact same question informative: [http://stackoverflow.com/q/4126874/272923](http://stackoverflow.com/q/4126874/272923)
Comment: @belisarius I wonder if the students fail for not displaying sufficient google-fu?
Comment: I wonder if some teacher out there is collecting Map clones
Comment: It's simpler to test something like this with a function that won't evaluate away, so you can really be sure you got the result you expect: `Map[f, {1,2,3}]` returns `{f[1],f[2],f[3]}`.
Answer: You're not passing s or x as variables, so it's just seeing s (which is an atomic expression) rather than a list.  You're definition needs to be mapAux[x_, s_, {}]:=..., which will make x and s take the values of the passed parameters.
Comment: Well you never actually apply x to First[s], so all it's doing is appending the first element of s to an empty list and returning that.  You're not doing any iterating or application.
Comment: Thanks! I fixed that, but now it's just outputting one of the list items such as {3} and it's not even adding it. Does the append need to be fixed?
Comment: @jackRAY, this has been answered before.  You should check out: http://stackoverflow.com/q/4126874/272923.


Iterate through a SC.ManyArrray to create custom views to append to another view in SproutCore

Question: I'm trying to figure out how I can retrieve a set of objects from the datastore, pass the SC.ManyArray to a controller then have a view iterate over the arrangedObjects to create custom SC.Views that I'd like to be able to append to other various views already appended to the DOM.
All I can find are examples of how to do this using the SproutCore pre built ListView/CollectionView/TableView etc. But I need these generated custom views to be appended to other views.
On another related but separate issue, when I inspect the object returned from the call to MyApp.store.find() the dataHashes property doesn't include all of my fixtures that I've included. I'm not sure if it's because I haven't created a view to call the data yet though because if I use the console I can return all the fixtures.
I'll also add that the data I'm retrieving is coming from a toMany relation from the master record. I'm retrieving the SC.ManyArray using a binding to the content of the SC.ArrayController with the path MyApp.masterModel.relationProperty.
Edit: I meant to say MyApp.masterModelController.relationProperty
Any insight would be great. I've been trying to figure this out for 2 days now and there doesn't seem to be any information found with google or the SproutCore docs.
Answer: I've figured it out...
What I ended up doing is creating a SC.CollectionView and bound the content to the SC.ArrayController used to managed the data collection. When I added this collection view to the childViews array, it automagically fetched the fixtures related to the models toMany relation and created the custom views I created and set as the exampleView on the SC.CollectionView and the stores dataHashes property was updated with these objects.
On the custom view I created to display the objects I overrode the didAppendToDocument method, acquired the path to the container view for the views I wanted to append these to and used appendChild to append the created custom views.
The last thing needed was to hide the SC.CollectionView so I just set the layout property to {top:-1000,left:-1000,width:0,height:0}.
I'm not 100% sure this is the best way to approach this but it does work (for me at least).
I'd love to hear any other suggestions to accomplish this.
Comment: To hide a view, all you need to do is update the `isVisible` property of the view to be `false` :-)