Renaming Rows in gtsummary, tbl_regression/tbl_stack

Question: I am using gtsummary and have stacked tables for the outputs of 11 different models on top of one another using the tbl_regression and tbl_stack functions. I am looking to see how the odds ratios of one consistent independent variable change across different dependent variables of interest. When I print out the tbl_stack output, it gives me 11 repeated rows of the independent variable name of interest, instead of the name of the unique dependent variables I am interested in. Does anyone know how to rename rows in tbl_stack (either manually or automatically) to put the dependent variable names in the rows?
Thank you.
Answer: You can use the tbl_regression(label=) argument to change the labels for the repeating variable to match the outcome for each model before you stack them.
There is, however, a shortcut using tbl_uvregerssion().
library(gtsummary)
packageVersion("gtsummary")
#> '1.3.6'

trial %>%
  select(response, death, age) %>%
  tbl_uvregression(
    x = age,
    method = glm,
    method.args = list(family = binomial),
    exponentiate = TRUE
  ) %>%
  modify_header(label ~ "**Outcome**")

Comment: Done. My rep score is still too low to have my upvote show up. And I am using survey data though so tbl_uvregression doesn't work. Thanks though.
Comment: Thanks! I used the label function to solve this (have covariates in my model). I have a follow up question that I've posted here that I would love your thoughts on!: https://stackoverflow.com/questions/65673290/merging-tbl-svysummary-and-stacked-tbl-regression-tables-with-different-variable
Comment: If they are the same covariates, you can use `tbl_uvregression(..., formula = "{y} ~ {x} + bmi")` to adjust each model for BMI. Only the x covariate will be shown.
Comment: if the post answers your question, please do mark it as correct and upvote...it'll help others find it in the future. @GMJ


Why is this threaded code not running concurrently?

Question: I'm attempting to thread some gui code to get rid of the lag in a combo box. The option the user selects does a bit of web scraping in the background, and depending on which option they select, the lag can sometimes be substantial as the program seeks the data it needs.
For the docs, it seems that to start a thread, I only need call threading.Thread with the appropriate arguments. So, my attempt of this in code is as follows:
def EvtComboBox(self, event):
    self.statusBox.AppendText('Fetching ' + event.GetString() + ' data.\n')
    loadingThread = threading.Thread(target=self.doStuff(event), name='i_am_a_thread')
    loadingThread.start()
    threading.Thread(target=self.checker(loadingThread)).start()

def doStuff(self, event):
    self.statusBox.AppendText(event.GetString() + ':\n')
    self.assets = self.scraper.loadConferencePage(event.GetString())
    self.statusBox.AppendText(str(self.scraper.getNumOfNewHandouts(self.assets)) + ' new handout(s) in Asset Manager\n' +
                                '-------------------------\n\n')

def checker(self, loadingThread):
    while threading.Thread.isAlive(loadingThread):
        print True

The behavior I'm expecting is that the loadingthread starts, and begins fetching the data it needs from the web. While it is doing that, I thought the second thread I create, would monitor that first thread, and my final assumption was that since those two threads are running "concurrently", the EvtComboBox method would finish, and the combobox in the gui would close.
However, none of that happens. The EvtComboBox method fires, and then the code seems to run completely sequentially. If I stick a print statement below the two threads (just to see when it gets executed,
def EvtComboBox(self, event):
    self.statusBox.AppendText('Fetching ' + event.GetString() + ' data.\n')
    loadingThread = threading.Thread(target=self.doStuff(event), name='i_am_a_thread')
    loadingThread.start()
    threading.Thread(target=self.checker(loadingThread)).start()
    print 'EvtComboBox method finished'

It only gets executed after the code the threads call gets run... so.. I'm unsure why it is behaving in this manner.
Why isn't it executing concurrently?
Answer: In the target part of threading.Thread you should not be calling the function, instead you give it the function object and arguments; such as threading.Thread(target=self.checker args=loadingThread).start()


How to get week dates in SQLite database

Question: I am developing android application. One of my application table has "foo_date" as column name. and data type is TEXT. I have following data in my table
primary key  | foo_date  | foo_value
--------------------------------------
1             2013-3-1    100

2             2013-3-2    2

3             2013-3-4    100

4             2013-3-3    1000

my requirement is to get data for the current week(i.e. 2013-3-4 to 2013-3-10). When i write below query, i am getting all data except row # 3
select * from foo_tbl where foo_date between '2013-3-4' and '2013-3-10';

How do I write sqlite query so that it includes all data for current week?
Thansk
Chintan
Comment: Storing week number also in DB would have helped a lot.
Comment: Can you post the query you are using at the moment?
Comment: i tried below queries but it doesn't give me the result i am looking for.
Comment: Query 1: select * from foo_tbl where foo_date >= '2013-3-4' and foo_date <= '2013-3-10';  Query 2: select * from foo_tbl where foo_date between '2013-3-4' and '2013-3-10'. None of these queires returns any record.
Comment: I did a similar thing. But I stored the day, month, year in separate columns. It made things easier. You may try that way if you wish.
Answer: Your problem is that you have stored the dates in a format that cannot be compared easily; the character 1 comes before 4, so 2013-3-10 compares less than 2013-3-4.
As documented, you should use the ISO date format yyyy-mm-dd. With constant field sizes, you can use a query with string comparisons that work:
SELECT * FROM foo WHERE foo_date BETWEEN '2013-03-04' AND '2013-03-10'

or use a date function to compute the week:
SELECT * FROM foo WHERE strftime('%W', foo_date) = '09'

Comment: Thank you very much for the help!! :) i changed format to "yyyy-mm-dd' and it's working now.
Answer: First, store the dates as standard sqlite time string format (i.e. YYYY-MM-DD).
Then, you can use this:
select * from foo where strftime('%W', foo_date) == strftime('%W', 'now')

to select the rows with the same week number as now.


How to prove the irrationality of this number.

Question: Prove that $\cos\frac{1}{\sqrt{2019}}$ is irrational.
Would someone give me a hand on this question? Even a hint is appreciated. I don't have any clue.
Comment: This follows from the [Lindemann-Weierstrass Theorem](https://en.wikipedia.org/wiki/Lindemann%E2%80%93Weierstrass_theorem)
Comment: See the answer of [this question](https://math.stackexchange.com/questions/1382606/when-cos-theta-1-8-its-easy-to-show-theta-is-an-irrational-angle-is/1382616#1382616).
Comment: Perhaps, you could use the Taylor series $\cos x = \sum_{k=0}^\infty \frac{(-1)^k x^{2k}}{(2k)!}.$
Comment: From here, a naïve guess would be to assume that the given number is rational, and derive a contradiction. If you need some inspiration, you might read this Wikipedia entry: https://en.wikipedia.org/wiki/Proof_that_e_is_irrational.
Comment: @Carlo, I have tried it, but it seems a dead end to me.
Comment: @Carlo, Thanks a lot!
Comment: $2019$ is square free and may be written as the product of $45 \pm \sqrt{6}$. Are these two numbers irrational? What happens when you take their square root? Can you then form partial fractions and use Trig sum angle formulae?
Comment: Neither prime numbers nor rational numbers come into play here.
Answer: Following the suggestions of @Carlo, I solved this problem myself. It is indeed similar to the proof that $e$ is irrational. Thinking it might help someone else, so I just post it in the following.
Suppose it is rational number, such that
$$\cos\frac{1}{\sqrt{2019}} = p/q,$$
where $p$ and $q$ are co-primes. Expand the cosine as sum of power series, which is
$$\cos\frac{1}{\sqrt{2019}} = \sum_{i=0}^\infty \frac{(-)^i}{(2i)!}\frac{1}{2019^i}.$$
Define the sum
$$S_n = \sum_{i=0}^nx_i,\quad \mathrm{and}\quad x_i=\frac{(-)^i}{(2i)!}\frac{1}{2019^i}$$
It is reasonable to assume for some integer large enough $N\gg1$, we have
$$x_{2m}+x_{2m+1}>0,\ \forall\ m\geqslant N,$$
also
$$x_{2m+1}+x_{2m+2}<0,\ \forall\ m\geqslant N,$$
So it is easy to see that
$$0<p/q-S_{2N-1}<x_{2N}=\frac{1}{(4N)!2019^{2N}}.$$
Thus if we choose, for example, that $4N>q$, we will have
$$0<(4N)!2019^{2N}\times(p/q-S_{2N-1})<1.$$
But on the other hand, $(4N)!2019^{2N}\times(p/q-S_{2N-1})$ must be an integer. Thus by contradiction, it must be irrational.
Comment: Yes. A little simpler: If $x=\cos 1/\sqrt {2019}\in \Bbb Q$ then $(2n)!x,$ and, a fortiori,  $(2019)^n(2n)!x,$ are integers for any sufficiently large $n,$ but that never happens.
Answer: If $x\in \Bbb R$ and for every $r>0$ there exists $<a,b>\in \Bbb Z\times \Bbb Z^+$ such that $0<|x-a/b|<r/b,$ then $x \not \in \Bbb Q.$
Proof: If $x=c/d$ with $c\in \Bbb Z$ and $d\in \Bbb Z^+$  then for any $<a,b>\in \Bbb Z\times \Bbb Z^+$ we have $$0<|x-a/b|\implies 0<|c/d-a/b|\implies 0<|cb-ad|\implies$$ $$\implies 1\le |cd-ab|\implies (1/d)/b\le (1/bd)|cb-ad|=|x-a/b|.$$ (I.e.  if $x=c/d$ then the "$r$" of my 1st sentence cannot be less than $1/d$.)
Apply this with $n\in \Bbb Z^+$ large enough that $r<(2n)!(2019)^n\
\cdot|\sum_{j=n+1}^{\infty}(-1)^j/((2j)!2019^j)\,|$
with $b=(2n)!(2019)^n$ and $a/b=\sum_{j=0}^n (-1)^j/((2j)!2019^j).$
Comment: Thank you for pointing out this theorem! It really simplifies the proof.
Comment: A common manoevre in the proof: Whenever you have an inequality $p<q$ between integers, it can be "sharpened" to $ p+1\le q.$


how to echo string from api json response with multi dimensional array

Question: I have a php script thats connected to an API and Im trying to format the response to look nice on the page. However I cant seem to access the data I need.
The json response looks something like this ---
[body] => stdClass Object
        (
            [items] => Array
                (
                    [0] => stdClass Object
                        (
                            [metadata] => Array
                                (
                                    [0] => stdClass Object
                                        (
                                            [key] => type
                                            [value] => challenge
                                        )

                                    [1] => stdClass Object
                                        (
                                            [key] => name
                                            [value] => do somethhing
                                        )

                                    [2] => stdClass Object
                                           .................
                                         )
                                      )
                     [1] => stdClass Object
                        (
                            [metadata] => Array
                                (
                                    [0] => stdClass Object
                                        (
                                            [key] => type
                                            [value] => challenge
                                        )

                                    [1] => stdClass Object
                                        (
                                            [key] => name
                                            [value] => do somethhing else
                                        )

                                    [2] => stdClass Object
                                    ................

I have tried the following --
//echo $response->raw_body->metadata->name;
//echo $response->raw_body->metadata;
//echo $response->raw_body;

$newresponse = $response->raw_body;
$items = json_decode($newresponse, true);

$response1 = json_decode($response, true);
echo $response1['body']['items']['metadata']['name'];

echo $items[2]['name'];
//echo $items->items;
//echo $items->items->metadata->name;
//echo $items->metadata->name;
echo $items->items[1]->metadata->name;

After I figure that, I am trying to loop the results in a for each loop -
foreach($items as $item) {
echo '
';
echo $item->items->metadata->name;
echo '
';
}

What am I doing wrong?
Answer: You have not name but key and value  so for json_decode as associative array
echo ['body']['items'][0]['metadata'][0]['key'];  // type

echo ['body']['items'][0]['metadata'][0]['value'];  // challenge

Answer: But inside of your metadata you have another array that you should iterate then echo only value
foreach($items as $item) {

$secondLoop=item->metadata;

foreach ($secondLoop as $loop)
{
    if($loop->key=="name")
        echo $loop->value;
 }
}

Ok (update answer) I try to make object array like you to test code
<?php

$c['body'] = (object)['items'=>[(object)["metadata"=>[(object)['key'=>'name','value'=>'cc']]]]];

var_dump($c);

$items=$c['body']->items;

foreach($items as $item) {

$secondLoop=$item->metadata;

foreach ($secondLoop as $loop)
{
    if($loop->key=="name")
        echo $loop->value;
 }
}

And the answer is
    array(1) {
  ["body"]=>
  object(stdClass)#3 (1) {
    ["items"]=>
    array(1) {
      [0]=>
      object(stdClass)#2 (1) {
        ["metadata"]=>
        array(1) {
          [0]=>
          object(stdClass)#1 (2) {
            ["key"]=>
            string(4) "name"
            ["value"]=>
            string(2) "cc"
          }
        }
      }
    }
  }
}
cc

Comment: Please first see this link :https://stackoverflow.com/questions/28608011/converting-mashape-api-response-to-php-string
Comment: Sorry but I don't understand how to apply this
Comment: Ok I edited the code, If you want to print value of array that contains only key with name index, you can use above code
Comment: i tried your code and got this error `Uncaught Error: Cannot use object of type stdClass as array in.. `
Comment: Ok I test the code, tell me if the your object is not as same as me
Comment: I received this error `Cannot use object of type Unirest\\Response as array in`


SNMP Trap passing notification to the script for further processing with NET-SNMP

Question: I have been trying to create a simple alert of interface down status  on dell s4810 switches by using NET-SNMP "traphandle". I was wondering how it could be possible to pass down notification information to the script so I could parse the text? I am able to activate the script but not pass down the notification. Perhaps there is a good python or perl module? I would be grateful for any advice or suggestion.
Answer: When you receive a SNMP trap, the daemon snmptrapd will forward this trap to an external program (typicaly your script).
snmptrapd will add some args (IP address, hostname, notification, OID) which will be sent in the standard input of your script.
An example of script which receive the trap, this script is from http://net-snmp.sourceforge.net/wiki/index.php/TUT:Configuring_snmptrapd:
<code>#!/bin/sh

 read host
 read ip
 vars=

 while read oid val
 do
   if [ "$vars" = "" ]
   then
     vars="$oid = $val"
   else
     vars="$vars, $oid = $val"
   fi
 done

 echo trap: $1 $host $ip $vars
</code>
This script will receive the args of the snmptrapd daemon, and will echo them . So you can add a command to export to a file to see if your configuration works fine.


EntityFramework 4, DbSet and ObjectContext

Question: few days ago i read tutorial about GenericRepository and Unit Of Work patterns http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application. I use web forms and i have EntityFramework CTP4 package installed. (I can't using EF 5).
I want to code generic repository for my project but i was stuck at this line:
this.dbSet = context.Set<TEntity>();

I know that this line doesn't work because a use ObjectContext in my project and database first approach. How can i deal with it? Can I code generic repository without migration to code first (which is not an option in my case) ?
Answer: This is the equivalent for ObjectContext:
this.dbSet = context.CreateObjectSet<TEntity>();

Now this creates an ObjectSet<TEntity> rather than a DbSet<TEntity>, but for your pattern you can use it in the same way.
UPDATE
The ObjectSet class does not have a utility method like that matches the Find() method of the DbSet.  In order to "Get by key" you would need to construct an EntityKey and use the ObjectContext.GetEntityByKey(), unfortunately that's not a really simple thing to do.
There really isn't a simple way to tackle this, that I've found.  In my case what I've done is to base all of my entities from a common class (using custom T4 templates to generate the classes from the model).  And then I can add a generic constraint to my repositories, like:
public class MyRepository<TEntity> where TEntity : MyEntityBaseClass

And then my common base class has an Id field which is inherited by all the entities so I can can simply do:
return myObjectSet.SingleOrDefault(x => x.Id == myId);

I'm sure there are other approaches, that might be a good topic for another question.
Comment: Thanks for help.

Now this creates an ObjectSet rather than a DbSet, but for your pattern you can use it in the same way.

I have problem with find method in this line:

    dbSet.Find(id);

How can i code this using ObjectSet ?
Comment: so if Find by Id in ObjectSet is so hard why you didn't use DbContext ? I wonder which of this method should i use, try to create find method as you or maybe i should use DbContext ? I haven't written to much code so far in this project, so maybe it's better idea to use DbContext ? Could you give your opinion about it ? Thanks
Comment: At the time that I started my application, EF Code First hadn't been released yet.  Based on Ropstah's answer it looks like there are templates you can use to generate Code First style classes for Model First or Database First applications.  I've never tried that approach, so I really cannot speak to whether or how well it works.
Answer: 1. You want to add the DbContextGenerator template to your visual studio templates:

2. After this make sure you clear out the default generation tool on your .edmx file.

3. Now you can implement the GenericRepository pattern as you wish.


Content API - parameter to find link url of a Link field

Question: I have a custom content type called Link, which has a Link field. I'm using services and Content API to query the list of Links, and I want to show the data for a Link node where the Link is equal to a parameter passed into the URL.
Eg, I would like to do something like the following:
[Drupal root]/api/content?field_link=http://www.google.com
Answer: The Content API is a tricky thing to get the hang of, but try to do the following:
As long as your Rest Server endpoint path is "api" and the resource is "content" what you have above should technically work.  Make sure you also enabled the Link content type in the Content API configuration.
When you do query with /api/content?field_link=http://www.google.com you will only get the base fields returned, even if you filtered by http://www.google.com it won't show in the response you get.  You then have to send another request to /api/content/ and that will return every field for that given node.
Hopefully this helps a little bit, if anyone can fill in my holes I would appreciate it, I'm by no means an expert, just played around with the Content API and got it to work a little.
I did respond to your email with some private links to prove that it is possible.
Comment: Thanks for this. I've just done a quick test, and it seems that if the query parameters don't find a match, then the whole result set is returned...It's annoying, but I can manage with that. Also, I wonder if anybody can find a way to filter the value of a Link content type.


A C# collection, which behaves like C++ set or priority_queue?

Question: I have written the Dijkstra's algorithm many times in C++ - I need there set or priotity_queue, both give me possibility to add an element and find the least one (using specified comparator). Now, I've got a problem when trying to write Dijkstra in C# - is there any structure which could be useful for me? I need adding and finding or erasing the least element.
Using Visual Studio '08
Answer: You can use SortedSet if you use a more recent .NET Framework version.


Visualizar imagens no banco de dados com arquivo separado do cakephp

Question: Estou usando o phpcake como administrador de uma galeria de fotos, as imagens sao salvas dentro da pasta webroot/img/galerias/ para cada categoria é criada uma pasta seguindo uma numeração. Ate ai tudo esta funcionando.
O problema é quando vou listar as imagens em um arquivo fora do phpcake só usando php puro, utilizo o seguinte codigo para chamar as imagens
<?php
    $photos = $conexao->query("SELECT * FROM images WHERE galeria_id = $album_id");
        while($photo_data = $photos->fetch_assoc()) { ?>

<img class="img-responsive" src=admin/webroot/img/galerias/"<?php echo $photo_data['galeria_id'] ?><?php echo $photo_data['image'] ?>" />

    <?php }
    ?>

Depois disso ele me retorna um erro no console do Chrome 403 (Forbidden) o no caminho da foto ele acrescenta %222bb1d11f-ea8a-4a27-b6f8-f28b1e5098ea.jpg%22
esses %222 no inicio e no fim da imagem. Eu só queria colocar o id que é igual ao numero da pasta e depois o caminho que esta salvo em imagem.
Comment: Consegui resolver
Answer: Coloca as aspas duplas no começo do caminho da imagem:
src="admin/webroot/img/galerias/`<?php echo $photo_data['galeria_id'] ?><?php echo $photo_data['image']` ?>"

E certifique-se que este é mesmo o endereço da imagem, pois tenho quase certeza que falta uma barra (/) entre os códigos php, e também tenho quase certeza que está faltando ou sobrando algo no começo do caminho...


Like Box not showing in some urls

Question: I have a simple yet hard problem here that is driving me nuts ...
Apparantly with no code change what so ever the Facebook and Google+ like boxes show as I want in one section of my site: http://www.nfrases.com/coragem
Yet on the other they just disappear, like on this page: http://www.nfrases.com/coragem/1519
The only difference is that a id_phrase is also suplied instead of only supplying the tag_name in the address that is picked by the php with $_GET and then the mod_rewrite intervenes. The actual address is: http://www.nfrases.com/tag.php?tag_nome=coragem&id_frase=1519
I'm going crazy over here! Someone ?
Answer: You are using a relative path for your javascript:

So, on http://www.nfrases.com/coragem, that would point to:
http://www.nfrases.com/js/scripts.js
But, on http://www.nfrases.com/coragem/1519, that would point to:
http://www.nfrases.com/coragem/js/scripts.js
The 2nd URL doesn't exist, obviously. You need to change your script path to:

Same rule applies for some of your images too. Example:

 Acerca

 Sugestões

 Contactos

Should be:

 Acerca

 Sugestões

 Contactos

Like Box not showing in some urls


Why does SAS PROC GLIMMIX give me VERY different random slopes than glmer (lme4) for a binomial glmm

Question: I am a user more familiar with R, and have been trying to estimate random slopes (selection coefficients) for about 35 individuals over 5 years for four habitat variables. The response variable is whether a location was "used" (1) or "available" (0) habitat ("use" below).
I am using a Windows 64-bit computer.
In R version 3.1.0, I use the data and expression below. PS, TH, RS, and HW are fixed effects (standardized, measured distance to habitat types). lme4 V 1.1-7.
str(dat)
'data.frame':   359756 obs. of  7 variables:
 $ use     : num  1 1 1 1 1 1 1 1 1 1 ...
 $ Year    : Factor w/ 5 levels "1","2","3","4",..: 4 4 4 4 4 4 4 4 3 4 ...
 $ ID      : num  306 306 306 306 306 306 306 306 162 306 ...
 $ PS: num  -0.32 -0.317 -0.317 -0.318 -0.317 ...
 $ TH: num  -0.211 -0.211 -0.211 -0.213 -0.22 ...
 $ RS: num  -0.337 -0.337 -0.337 -0.337 -0.337 ...
 $ HW: num  -0.0258 -0.19 -0.19 -0.19 -0.4561 ...

glmer(use ~  PS + TH + RS + HW +
     (1 + PS + TH + RS + HW |ID/Year),
     family = binomial, data = dat, control=glmerControl(optimizer="bobyqa"))

glmer gives me parameter estimates for the fixed effects that make sense to me, and the random slopes (which I interpret as selection coefficients to each habitat type) also make sense when I investigate the data qualitatively. The log-likelihood for the model is -3050.8.
However, most research in animal ecology do not use R because with animal location data, spatial autocorrelation can make the standard errors prone to type I error. While R uses model-based standard errors, empirical (also Huber-white or sandwich) standard errors are preferred.
While R does not currently offer this option (to my knowledge - PLEASE, correct me if I am wrong), SAS does - although I do not have access to SAS, a colleague agreed to let me borrow his computer to determine if the standard errors change significantly when the empirical method is used.
First, we wished to ensure that when using model-based standard errors, SAS would produce similar estimates to R - to be certain that the model is specified the same way in both programs. I don't care if they are exactly the same - just similar.
I tried (SAS V 9.2):
proc glimmix data=dat method=laplace;
   class year id;
   model use =  PS TH RS HW / dist=bin solution ddfm=betwithin;
   random intercept PS TH RS HW / subject = year(id) solution type=UN;
run;title;

I also tried various other forms, such as adding lines
random intercept / subject = year(id) solution type=UN;
random intercept PS TH RS HW / subject = id solution type=UN;

I tried without specifying the
solution type = UN,

or commenting out
ddfm=betwithin;

No matter how we specify the model (and we have tried many ways), I cannot get the random slopes in SAS to remotely resemble those output from R - even though the fixed effects are similar enough. And when I mean different, I mean that not even the signs are the same. The -2 Log Likelihood in SAS was 71344.94.
I can't upload my full dataset; so I made a toy dataset with only the records from three individuals. SAS gives me output in a few minutes; in R it takes over an hour. Weird. With this toy dataset I'm now getting different estimates for the fixed effects.
My question:
Can anyone shed light on why the random slopes estimates might be so different between R and SAS? Is there anything I can do in R, or SAS, to modify my code so that the calls produce similar results? I'd rather change the code in SAS, since I "believe" my R estimates more.
I'm really concerned with these differences and want to get to the bottom of this problem!
My output from a toy dataset that uses only three of the 35 individuals in the full dataset for R and SAS are included as jpegs.

EDIT AND UPDATE:
As @JakeWestfall helped discover, the slopes in SAS do not include the fixed effects. When I add the fixed effects, here is the result - comparing R slopes to SAS slopes for one fixed effect, "PS", between programs: (Selection coefficient = random slope). Note the increased variation in SAS.
Comment: Have you checked that you are modeling the dependent variable in the same direction?
Comment: By the way, what Peter is getting at is that, by default with binomial data labeled as `0`s and `1`s, `R` will model the probability of a "1" response while SAS will model the probability of a "0" response. To make SAS model the probability of "1" you need to write your response variable as `use(event='1')`. Of course, even without doing this I believe we should still expect the same estimates of the random effect variances, as well as the same fixed effect estimates albeit with their signs reversed.
Comment: @EricaN One thing that you just reminded me of is that you should compare the random effects from R to those in SAS using the `ranef()` function rather than `coef()`. The former gives the actual random effects, while the latter gives the random effects plus the fixed-effects vector. So this accounts for a lot of why the numbers illustrated in your post differ, but there is still a substantial discrepancy remaining that I can't totally explain.
Comment: I notice that `ID` isn't a factor in R; check and see if that changes anything.
Comment: I see that you are fitting both using Laplace Approximation for the log-likelihood. What are their respective log-likelihood scores?
Comment: I don't see anything obvious.  `ID` should get converted to a factor when it's used as a grouping variable.  I'm not sure whether the nesting syntax is the same: `ID/Year` means "year nested within ID", which seems reasonable; is this equivalent to the SAS spec? You may send me data if you like ...
Comment: **@usεr11852**, I edited my post to include the loglik and -2Log Likelihood results from R and SAS. **@Ben Bolker**, according to the [SAS help menu](http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_anova_sect017.htm), the nested random effect comes first, followed by the main random effect in parentheses - so my syntax should match that in R. @Peter Flom - not sure what you mean?
Comment: I can verify that the two nesting syntaxes should be equivalent. @EricaN I think you really need to paste the full output from both `SAS` and `R`. (I notice for example that the log-likelihoods appear to be on radically different scales, is this perhaps a typo or pasting error?) Even better would be an actual reproducible example, but maybe we can start with the full outputs.
Comment: @JakeWestfall, I added the output from my toy dataset (so that if I upload it we are both working from the same dataset). What is the recommended repository if I choose to share a .csv?
Comment: The correlation between your random effects is pretty high. That can sometimes lead to computational issues, if so, centering the variables can sometimes help.
Comment: Also note the binned residual plots are disturbing, so maybe these data need more than a little bit of tweaking...
Comment: As @Jake mentions these log-likelihoods are vastly different so it will not be surprising for something fishy to be going on numerically. Can you check what results you will you get if instead of BOBYQA for the optimization you use `method="nlminb"` or `method="L-BFGS-B"`? Do they converge to approximately the same solution? Finally, numerics aside, are you able to interpreter the coefficients? Do they appear sensible based on your knowledge of the problem? Eg. the variable `RS` switches signs between the runs, any idea what the sign might be?
Comment: It's ok that the sign for RS switches between runs I think... there is not a strong signal of this habitat type, so I wouldn't expect the results for that one to be consistent. However, I DO expect that most of the random slopes for HW to all be negative. Perhaps I should try to log transform the data and try again, with your suggestions @usεr11852. Unfortunately, a "try" takes over 10 hours with glmer...
Comment: @Aaron, I should mention that the variables have been standardized by subtracting the mean and dividing by two standard deviations.
Comment: @JakeWestfall, I just noticed that the random slopes from my model in glmer are very similar to those output by SAS when I _do not_ specify fixed effects in glmer - I can see that SAS is computing the estimates for fixed effects, but perhaps this is a clue? I'm still trying to get this to work...
Comment: @JakeWestfall we are getting closer to an answer here... so if I "add" the fixed effects to each random slope in SAS, they should output very similar answers. Even though the _trend_ is now more similar, you are right - there is still a lot of discrepancy.
Comment: @JakeWestfall, do you know how to ask SAS for the random slopes with fixed effects added or do you do it manually?
Comment: @EricaN I did it manually by adding -1.87 to the random effects.
Comment: I think SAS gives the deviance estimate (Difference: fixed effect model-Random effect model).
Answer: It appears that I shouldn't have expected the random slopes to be similar between packages, according to Zhang et al 2011. In their paper On Fitting Generalized Linear Mixed-effects Models for Binary Responses using Different Statistical Packages, they describe:
Abstract:

The generalized linear mixed-effects model (GLMM) is a popular paradigm to extend models for cross-sectional data to a longitudinal setting. When applied to modeling binary responses, different software packages and even different procedures within a package may give quite different results. In this report, we describe the statistical approaches that underlie these different procedures and discuss their strengths and weaknesses when applied to fit correlated binary responses. We then illustrate these considerations by applying these procedures implemented in some popular software packages to simulated and real study data. Our simulation results indicate a lack of reliability for most of the procedures considered, which carries significant implications for applying such popular software packages in practice.

I hope @BenBolker and team will consider my question as a vote for R to incorporate empirical standard errors and Gauss-Hermite Quadrature capability for models with several random slope terms to glmer, as I prefer the R interface and would love to be able to apply some further analyses in that program. Happily, even while R and SAS do not have comparable values for random slopes, the overall trends are the same. Thanks all for your input, I really appreciate the time and consideration that you put into this!
Comment: Nope. I keep trying to figure out how I'm going to support development like this, since it's not technically part of my research program ...
Comment: sorry: what's a "standard standard error"?  Do you mean standard errors of variance components?  Or do you mean sandwich standard errors?
Comment: sorry... meant empirical/sandwich SEs. I've edited my response.
Comment: @BenBolker Did this ever get incorporated?
Answer: A mixture of an answer and commentary/more questions:
I fitted the 'toy' data set with three different optimizer choices. (*Note 1: it would probably be more useful for comparative purposes to make a small data set by subsampling from within every year and id, rather than by subsampling the grouping variables.  As it is, we know that the GLMM won't perform particularly well with such a small number of grouping variable levels.  You can do this via something like:
library(plyr)
subdata <- ddply(fulldata,c("year","id"),
    function(x) x[sample(nrow(x),size=round(nrow(x)*0.1)),])

Batch fitting code:
Ntoy <- readRDS("Newton_toy.RDS")
library(lme4)
fitfun <- function(opt) {
    tt <- system.time(fit1 <- glmer(use ~  ps + th + rs + hw +
                                    (1 + ps + th + rs + hw |id/year),
                                    family = binomial, data = Ntoy,
                                    control=glmerControl(optimizer=opt),
                                    verbose=100))
    return(list(time=tt,fit=fit1))
}

opts <- c("nloptwrap","nlminbwrap","bobyqa")
## use for() instead of lapply so we can checkpoint more easily
res <- setNames(vector("list",length(opts)),opts)
for (i in opts) {
    res[[i]] <- fitfun(i)
    save("res",file="Newton_batch.RData")
}

Then I read in the results in a new session:
load("Newton_batch.RData")
library(lme4)

Elapsed time and deviance:
cbind(time=unname(sapply(res,function(x) x$time["elapsed"])),
          dev=sapply(res,function(x) deviance(x$fit)))
##                time      dev
## nloptwrap  1001.824 6067.706
## nlminbwrap 3495.671 6068.730
## bobyqa     4945.332 6068.731

These deviances are considerably below the deviance reported by the OP from R (6101.7), and slightly below the ones reported by the OP from SAS (6078.9), although comparing deviances across packages is not always sensible.
I was indeed surprised that SAS converged in only about 100 function evaluations!
Times range from 17 minutes (nloptwrap) to 80 minutes (bobyqa) on a Macbook Pro, consistent with the OP's experience.  Deviance is a bit better for nloptwrap.
round(cbind(sapply(res,function(x) fixef(x$fit))),3)
##             nloptwrap nlminbwrap bobyqa
## (Intercept)    -5.815     -5.322 -5.322
## ps             -0.989      0.171  0.171
## th             -0.033     -1.342 -1.341
## rs              1.361     -0.140 -0.139
## hw             -2.100     -2.082 -2.082

Answers appear quite different with nloptwrap -- although the standard errors are quite large ...
round(coef(summary(res[[1]]$fit)),3)
##             Estimate Std. Error z value Pr(>|z|)
## (Intercept)   -5.815      0.750  -7.750    0.000
## ps            -0.989      1.275  -0.776    0.438
## th            -0.033      2.482  -0.013    0.989
## rs             1.361      2.799   0.486    0.627
## hw            -2.100      0.490  -4.283    0.000

(code here gives some warnings about year:id that I should track down)
To be continued ... ?
Comment: would it be more helpful if I sent you the full dataset? The only problem is that convergence takes about 9 hrs with the full dataset, so your suggestion regarding sampling is a good one. I tried to transform the data with a log transformation, but the binned residual plot is still ugly - do you think the residual plot explains part of the problem with these data? Finally - were your results in SAS similar to R's?


Handling an embedded view inside a ListView?

Question: I have a quite problematic UI layout to implement, and I'm not sure if it's even possible using standard UI widgets. It looks something like this:
Picture 1
The green, lined thing is supposed to be a ListView, and the red rectangle is another View. This red View should be scrolled with the ListView, as if it's part of it. Some list-elements should also be narrower, because that embedded View gets in their way. Could you please recommend any ideas? Can this be done somehow with the Android UI framework? I was thinking about some kind of a floating View above the ListView, which reacts to the List's scrolling events too, but it doesn't seem like an elegant solution.
Thanks
Answer: I don't think you can accomplish that easily with a ListView.  You could do the overlay using a FrameLayout, but it would be very awkward to get it to stay probably aligned as the user scrolls.
How many elements are you talking about?
I would probably use a LinearLayout within a ScrollPane to simulate the ListView.
Or, a TableLayout where the overlayed view is contained within a single, complex row.
Answer: I would set the green rows that the red block overlap and the red block as one big view in the listview.  So the items in your listview would be (for the example pic) two green rows, then the view of three green rows and the overlapping red block, and then the remainder of the green rows.
Trying to have the red block on an overlay that scrolls with the listview sounds like more trouble than it's worth.
Comment: That's an interesting idea, although I'd have to hack a quite weird adapter for the list for that. I'll try it out, thanks.
Comment: Ok, I got it with this solution, it required a rather ugly adapter tho. Thanks!


How to fix the "Not Acceptable 406" error with arabic language in url?

Question: How to fix the "Not Acceptable 406" error with arabic language in url ?

Not Acceptable
An appropriate representation of the
  requested resource / could not be
  found on this server.
Additionally, a 404 Not Found error
  was encountered while trying to use an
  ErrorDocument to handle the reque

The error doesn't appear when I put english words in the url :
http://www.faressoft.org/?anything=hello
but it appears when I put arabic words in the url :
http://www.faressoft.org/?anything=%E3%D1%CD%C8%C7%F0
I have this problem in all pages.
Comment: I guess that the error is not caused by apache but by the script that is handling the request (and parsing the querystring parameters).
Comment: but I have this problem in all pages.
Answer: Apache is probably configured to the character set UTF-8. But the end of your URL decodes to the byte sequence E3D1CDC8C7F0, which is not a valid UTF-8 byte sequence. That's probably why Apache raises an error.
In what character set is E3D1CDC8C7F0 a valid byte sequence? What information does the HTTP header contain about the character set (you can use the HTTP Live Header plug-in for Firefox to find out)?
Update:
Obviously, my answer got downvoted (without any comments). However, the accepted answer and my answer are related. My answer gives the technical details and the accepted answer explains which module is responsible for the check.
I still don't understand how the URL in your question was created in the first place. I don't think it's a valid URL.
I've copied some arabic text from your web site, URL encoded it and created the following URL, which works:
http://www.faressoft.org/?anything=%D9%87%D9%86%D8%A7%D9%83
When the page is displayed, the URL is even displayed with arabic characters in Firefox.
Are you sure, disabling mod_security will solve your problem? What if your URL is wrong?
P.S.: As I don't understand any Arabic I hope I didn't copy any text that is insulting or confusing in this new context.
Answer: Hm, this can be a lot of things. First of all, do you have mod_security? That's an Apache module supposedly making your site more secure however without a real lot of understanding of it just breaks your site. Just switch it off.
Comment: Yes, My hoster has mod_security in his hosting. How can I disabled it (I have a shared hosting not a server).
Comment: I guess this answer and my answer are related. And disabling mod_security might not solve your problem. See my update.


fetch specifi data from json and store them in array using node js / Javascript

Question: I am trying to fetch all the log details from this json object but the problem is this key always get changed
let userJsonObject =
{
  "user_1":{
     "id":"user_1",
     "log":"de-data",
     "name":"dummy test",
     "address":"New York"
  },
  "user_2":{
     "id":"user_2",
     "log":"ex-file",
     "name":"tom",
     "address":"Nevada"
  },
  "user_3":{
     "id":"user_3",
     "log":"dis-acta",
     "name":"Jerry",
     "address":"LA"
  },
}

this is my json Object and I want fetch all the log which are present inside the json object and store them in one array like this
let gotLog= ["de-data","ex-file","dis-acta"]

here user_1 user_2 get changed everytime and I wanted to tackle this scenario and fetch all the log in one array
Comment: user_1 changes to x name, and you don't know it? did I understood well?
Answer: var obj = { first: 'someVal' };
obj[Object.keys(obj)[0]];

I found this here How to access the first property of a Javascript object?
Answer:

let userJsonObject =
{
  "user_1":{
     "id":"user_1",
     "log":"de-data",
     "name":"dummy test",
     "address":"New York"
  },
  "user_2":{
     "id":"user_2",
     "log":"ex-file",
     "name":"tom",
     "address":"Nevada"
  },
  "user_3":{
     "id":"user_3",
     "log":"dis-acta",
     "name":"Jerry",
     "address":"LA"
  },
};

let array=[];
for(const value of Object.values(userJsonObject)){
    array.push(value.log)
};
console.log(array);

I got it :)


Remove trailing slash at the end of axios url request

Question: I am trying to do a request with axios, but it keeps adding a trailing slash right before my query parameters and so it stops working, I can't figure out for the life of me what the hell is its problem and I cant find any options in the config to remove it.
This is the url that I want to send request to
http://localhost:3000/api/competition/player-joined?userId=${accounts[i].dbId}&competitionId=a80flwvap1lbhef3d704
Axios sends request to this instead:
http://localhost:3000/api/competition/player-joined/?userId=${accounts[i].dbId}&competitionId=a80flwvap1lbhef3d704
There is a / right before the ? being added by axios itself when sending the request.
This is a standalone js file that I am running with node.
The suggested questions that supposedly solve this issue DO NOT SOLVE THE ISSUE.
Also, Axios is automatically doing that, I am not giving it a string with a slash, so its not a matter of simple "replace string".
Comment: Why would you close my question when what you are suggesting has nothing to do with my problem?
Comment: @Vanojx1 no it does not, I already saw this post.
Comment: Voted to reopen, please add some reference code.
Comment: Sounds more like url rewriting on your server is causing this. If not, ditch axios and use [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).
Comment: @Besworks that turned out to be correct, I did manage to solve it a week ago when I had the issue. The problem turned out to be even more stupid than this (because I also did try fetch, it was behaving the same way), my URL was wrong, there was one more argument missing to make the URL valid, so yes. You are correct in your assumption and I was simply wrong with the URL itself. This is a Nextjs project, initially I thought its caused by Axios.
Answer: The answer turned out to be very simple.
As pointed the suggested answers were not helping, the problem was in an entirely different place and not with axios.
As I was testing I had also tried it with fetch but it was doing the same thing, and at some point I decided to try Postman, and it too was not working! But I knew that the URL itself is working, because I am using it in multiple places within my app and they all were working, except this new thing that I was developing, which eventually helped me find the problem.
The problem was that my URL itself was not valid!
http://localhost:3000/api/competition/player-joined?...
was supposed to be
http://localhost:3000/api/competition/update/player-joined?...
And as @Besworks suggested in a comment above, it was the server rewriting the URL.
The moment I fixed the URL, everything worked just fine.


Laurent series and residue of $f(x)=\frac{1}{1+e^z}$

Question: I am having trouble trying to expand this function using Laurent series, and finding the residue$$f(x)=\frac{1}{1+e^z}$$
If I replace $e^z$ with its series I get $$f(x)=\frac{1}{1+\sum_{n=0}^\infty\frac{z^n}{n!}}$$
and the expansion is $$\frac{1}{2}+\frac{1}{1+z}+\frac{1}{1+\frac{z^2}{n!}}+...$$ which is not useful in finding the residue at the singularity point(s) $z_0=(2k+1)\pi i$.
What would be a proper expansion, or is there another way to solve this?
edit:fixed summation formula layout
Comment: What do you want exactly? The laurent series? The residue? Both? A closed formula for the laurent series or just a few terms?
Comment: I need to find the residue, and for that i thought I need a couple of terms from the expansion. The terms are not really necessary, if there is another way to find the residue.
Comment: [This](http://math.stackexchange.com/a/804627/55235) answer of mine should help you. The first part of the answer gives you a formula to find the residue. The second part hints at how you can find the laurent series to find the residue.
Answer: You can use the formula for finding residues of a pole $z_0$ of order $1$:
$$\text{Res}(f, z_0)=\lim \limits_{z\to z_0}\left[(z-z_0)f(z)\right].$$
The singularities here are the points $(2k+1)\pi i$, where $k$ ranges over the integers.
To use the above formula you need to first prove that these points are poles of order $1$, that is, you need to prove that these points are not removable discontinuities and you need to prove that $\lim \limits_{z\to (2k+1)\pi i}\left(\dfrac{z-(2k+1)\pi i}{e^z+1}\right)\in \mathbb C$. This limit actually is the same as in the above formula, so if you find it, you immediately find your residue.
One has
$$\begin{align} \lim \limits_{z\to (2k+1)\pi i}\left(\dfrac{z-(2k+1)\pi i}{e^z+1}\right)&=\lim \limits_{w\to 0}\left(\dfrac{w}{e^{w+(2k+1)\pi i}+1}\right)\\
&=\lim \limits_{w\to 0}\left(\dfrac{w}{e^{w+\pi i}+1}\right)\\
&=\lim \limits_{w\to 0}\left(\dfrac{1}{e^{w+\pi i}}\right)\\
&=-1.
\end{align}$$
Hence $\forall k\in \mathbb Z\left(\text{Res}(f,(2k+1)\pi i)=-1\right)$ which agrees with Wolfram Alpha.
In the above I used Cauchy's rule. If you want to avoid without having to resort to Laurent series, you can first prove that $\lim \limits_{w\to 0}\left(\dfrac{e^{w+\pi i}+1}w\right)=1$ and this you can do with the Taylor Series.
Comment: Thank you for the help, but i still have one small issue, and this has been my problem from the start. How exactly do you prove that those singularity points are poles of order 1?
Comment: I addressed that issue in my answer "You need to prove that these points are not removable discontinuities and you need to prove that $\lim \limits_{z\to (2k+1)\pi i}\left(\dfrac{z-(2k+1)\pi i}{e^z+1}\right)\in \mathbb C$". I took care of the limit, you just have to prove that they aren't removable discontinuities.
Answer: As Git explained, you need to calculate
$$\lim_{z\to (2k+1)\pi i}\dfrac{z-(2k+1)\pi i}{e^z+1} \,.$$
The fastest way of doing this is to observe that
$$\lim_{z\to (2k+1)\pi i}\dfrac{e^z+1}{z-(2k+1)\pi i} \,.$$
is just the definition of the derivative of $e^z$ at $z=(2k+1)\pi i$. Thus
$$\lim_{z\to (2k+1)\pi i}\dfrac{e^z+1}{z-(2k+1)\pi i}=e^{(2k+1)\pi i}=-1$$
Comment: Nice. I usually am on the look out for these tricks. I missed it.


ERR_PNPM_ERR_PNPM_UNEXPECTED_PKG_CONTENT_IN_STORE The lockfile is broken! A full installation will be performed in an attempt to fix it

Question: enter image description here
ERR_PNPM_ERR_PNPM_UNEXPECTED_PKG_CONTENT_IN_STORE
Answer: There was an edge case in an older version of pnpm (I don't remember exactly which, probably pre v6.30), where the store got corrupted. So I would suggest to

upgrade pnpm to the latest version
remove the store (to get the store location, run pnpm store path)
run pnpm install

If the lockfile is changed, it means the lockfile was broken, so commit the changed lockfile.
Answer: I encountered this problem not long ago on firstname.lastname@example.com.
I finally solved it via rm -rf $(pnpm store path).


Okhttp is quietly retrying requests, is there a way to log the retries?

Question: We've been using okhttpClient in our android app, and noticed that OkHttpClient lib is quietly retrying requests, with the config retryOnConnectionFailure boolean in OkHttpClient class.
Is there any way to log the retries? is there a callback or lambda we can use?
I looked into the source code of okhttpClient and did some research, but was not lucky, didn't find anything.
Comment: Please provide enough code so others can better understand or reproduce the problem.
Answer: You log requests via interceptor.
in your *.gradle add
implementation 'com.squareup.okhttp3:logging-interceptor:4.9.1'

and to your OK client:
httpClient.addNetworkInterceptor(HttpLoggingInterceptor().setLevel(HttpLoggingInterceptor.Level.BODY))

Now, there are 2 places to hook an interceptor, addInterceptor and addNetworkInterceptor. The difference is that regular interceptor shows all requests, including ones served from local cache and network interceptors shows only traffic to a server.
Comment: Thanks. this is great
Is there any easy way to know the retry count of a specific http call, like something we can use in the request or response header??
Comment: @Xiaoqian I don't know about any in particular. You might try writing your own interceptor, maybe there's something there.


Login Simultâneo banco de dados - Eficiência

Question: Olá!
Para proibir login simultâneo de um mesmo usuário em um sistema local PHP, onde possuirá no máximo 50 usuários a longo prazo (inicialmente entre 15 e 20), criei uma tabela no banco de dados, aonde dados de uma sessão é inserida a cada vez que o usuário faz login nas colunas (id,id_usuario,created), sendo que o sistema faz a verificação se há alguma ativa com o ID do usuário. Caso encontre é redirecionado para uma página aonde pode optar por destruir todas as sessões ou manter a sessão ativa, em que nos dois casos retorna para a página de login, aonde na primeira opção passa por uma página de logout antes apagando registro no banco de dados (apagando registro no banco de dados, que mediante uma nova requisição do usuário "antigo" também será redirecionado para página de login. Atualizar a página não derruba ele), e na segunda não acontece nada (apenas o redirecionamento e mantendo registro da sessão do usuário antigo). Também fiz pela tag do HTML5 o redirecionamento em caso de inatividade por 1hr.
Uma das coisas que confesso que estou incomodado é o fato de a cada requisição de página ter de haver a consulta ao banco de dados para saber se a sessão está ativa, fazendo uma instanciação a cada requisição, tenho receio de que isso afete o desempenho...
1 - Gostaria de saber de vocês se acham essa solução eficiente para um sistema local?
2 - Percebi que o SQL, a cada nova inserção no BD mesmo que não tenha nenhum registro na tabela, ele incrementa o id da sessão de onde parou na inserção anterior... sei que o tipo INT tem um limite, quando for atingido ele voltará a incrementar a partir do 1 ou terei de mudar de tipo?
Answer: Para validar se não existem múltiplos acessos com o mesmo usuário, apenas verifique no login se o usuário já tem uma sessão aberta, se houver trate como já faz, senão acesse normalmente. O usuário não tem como acessar outras paginas sem passar pelo login, então não precisa checar a cada requisição, somente no login.
Agora o campo INT armazena até 2.147.483.647, pela quantidade de usuários utilizando o sistema, você pode ficar tranquilo que suporta. Mas faça uma conta para validar.
Quantidade de logins diarios. Ex.: 50 usuarios x 100 logins = 5.000, acho muito essa quantidade de logins, mas estou exagerando bastante. O INT vai até 2.147.483.647 então 2.147.483.647/5000 = 429.496 dias e 429.496 / 365 = 1.176 anos. Altere os valores de acessos para os reais e veja quanto tempo você levaria para preencher todos.
Obs.: Essa conta é para ilustrar o quanto é grande o valor do INT, mas outros fatores podem influenciar na escolha do tipo.
Comment: Obrigado pela dica!
Obs.: notei que quando desligo o note e não tenha ficado nenhuma sessão ativa, ele volta a inserir do ID 1 novamente!
Com relação a checar a cada requisição a lógica é a seguinte: checa no login pra ver se possui sessão ativa para o usuário, se sim redireciona para derrubar ou manter sessão antiga, e a cada requisição de página para ver se possui sessão ativa (comparando login e id da sessão) para que ele não possa continuar navegando normalmente depois de "derrubado", sendo que haverá outra sessão com seu login, o que dará true, mas por consultar o ID da sessão será F


Guarda de rotas no Ionic é possível?

Question: Atualmente estou fazendo a segurança da minhas rotas no ngOnInit, se tiver no localStorage a chave token, então ele permanece na rota Home (Minha rota root é a home). Caso contrário, ele redireciona para a tela de login.
O problema é que tem um delay, a tela Home é mostrada durante alguns milisegundos e depois troca para a tela de Login.
Já tentei no constructor/ngOnInit e também no appcomponen.ts assim que a plataforma é pronta, porém dessa maneira eu continuo tendo uma "piscada" na tela.
Alguém tem alguma solução para isso?
Comment: E possivel da uma olhada na documentacao: https://angular.io/guide/router#milestone-5-route-guards
Comment: Mas isso é aplicável no Ionic?
Comment: Porque supostamente o Ionic não tem rotas...
Answer: [EDITADO] - Fiz em ionic 3 para voce. Faz isso no seu app.component.ts
Eu estou usando Ionic 4, no meu arquivo app-routing.module.ts eu uso em Guarda de Rotas para verificar se o usuário tem alguma informação salva no local storage, caso tenha eu redireciono para a home ou se não tiver eu mando ele para login. Segue a logica.
import { Component, ViewChild } from '@angular/core';
import { Platform, Nav, ToastController } from 'ionic-angular';
import { StatusBar } from '@ionic-native/status-bar';
import { SplashScreen } from '@ionic-native/splash-screen';
import { SharedData } from '../providers/shared-data';
import { ClientStorage } from '../database/client-storage';

@Component({
    templateUrl: 'app.html'
})
export class MyApp {
    @ViewChild(Nav) nav: Nav;
    rootPage: any;

    constructor(
        public platform: Platform,
        public statusBar: StatusBar,
        public splashScreen: SplashScreen,
        public sharedData: SharedData,
        private clientStorage: ClientStorage,
    ) {
            this.clientStorage.buscaRealizada.then(() => {
                statusBar.styleDefault();
                splashScreen.hide();
                if (this.clientStorage.data.UserId)
                    this.rootPage = 'HomePage';
                else
                    this.rootPage = 'LoginPage';
            });
        });
    }
}

Comment: a pergunta era referente ao ionic v3
Comment: Olha esse link então, é em Ionic 2 poré, vc consegue adaptar para o 3 de uma forma mais fácil -> https://medium.com/lfdev-blog/guarda-de-rota-em-ionic-2-1dd3451839b3
Comment: O certa mesmo, e você fazer essa parte de acesso a rotas no app.component.ts, pq antes de iniciar o app ele ja vai mapear isso e redirecionar para local certo. E dai não vai ter esse delay
Comment: Editado para Ionic 3
Answer: Sim, existe outra solução melhor. Você pode dar uma lida nos Lifecycle events na documentação do IONIC. No seu caso, vc pode usar o ionViewCanEnter. Antes da página ser criada, ele vai verificar sua lógica. Se o usuario tiver acesso, sua logica retorna true e cria a pagina a ser acessada, se não retorna false.
Veja este tutorial
import { Component } from '@angular/core';
import { NavController, NavParams } from 'ionic-angular';

@Component({
  selector: 'page-a',
  templateUrl: 'a.html'
})
export class PageA{

  constructor(public navCtrl: NavController) {}

  ionViewCanEnter() {
    \\sua logica aqui. retorne true ou false.
  }
}

Comment: Mesmo usando ionViewCanEnter a tela ainda aparece, mesmo que em milisegundos.

Guarda de rotas no Ionic é possível?


Changing a plain text field to enable rich editing

Question: In Drupal 8, I set up a plain text field and now I have no way to change it back to rich text editing.
I could take the approach of just setting up a new field and copy the field data from the old one to the new one manually, but I have hundreds of records. What are some other solutions?
Answer: There is no official solution. String and text fields have different structure (text has an additional format column), you could try to convert it by hand, but you also need to update Drupal's internal information about the field and resulting database schema.
Your easiest option is probably to create a new field and then write a simple script that loads all nodes of that type and assigns the value of the old field to the new. If there are too many nodes to load for a single run, you'll need to split it up or do a batch.
<code>$nodes = \Drupal::entityTypeManager()->getStorage('node')->loadByProperties(['type' => 'yourtype']);
foreach ($nodes as $node) {
  $node->newfield->value = $node->oldfield->value;
  $node->newfield->format = 'plain_text';
  $node->save();
}
</code>
Comment: 'plain_text' rather than 'text_plain'
Comment: Where can I use this script? In a hook ?


Trying to connect to an LDAP server with TLS using python-ldap module

Question: Connection to LDAP server fails through TLS connection
I am using Python 2.7 ldap module, and have tried connecting to an LDAP server with TLS enabled, but so far I have only run into many issues. When trying to debug the issue I get very little information back. Here is a simple script that I am testing with below
import ldap

LDAP_SERVER = 'ldap://ldap.somedomain.com:389'
LDAP_BASE = 'ou=users,dc=ldap,dc=test,dc=com'

try:
    conn = ldap.initialize(LDAP_SERVER, bytes_mode=False)
    conn.set_option(ldap.OPT_REFERRALS, 0)
    conn.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
    conn.set_option(ldap.OPT_X_TLS_CACERTFILE, "/path/to/cacert.pem")
    conn.set_option(ldap.OPT_X_TLS,ldap.OPT_X_TLS_DEMAND)
    conn.set_option(ldap.OPT_X_TLS_DEMAND, True)
    conn.set_option(ldap.OPT_DEBUG_LEVEL, 255)
    conn.start_tls_s()
except ldap.LDAPError, e:
    print e
    raise

print 'done'

When testing the script, an Exception is raised when the conn.start_tls_s line is executed. here is the error that is returned:
ldap.CONNECT_ERROR: {'info': u'(unknown error code)', 'errno': 2, 'desc': u'Connect error'}
Stack Trace:
File "/home/eric/Desktop/test_ldap.py", line 14, in <module>
    conn.start_tls_s()
  File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 864, in start_tls_s
    return self._ldap_call(self._l.start_tls_s)
  File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 329, in _ldap_call
    reraise(exc_type, exc_value, exc_traceback)
  File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call
    result = func(*args,**kwargs)

Note - Port 389 on my LDAP server is secure and accepts TLS connections, which is why the LDAP_SERVER is not set to  ldaps://ldap.somedomain.com:636 in the example above.
I am also using an Ubuntu 14.04 virtual machine. Any ideas, and tips are greatly appreciated.
Comment: Did you try from the command line from same host like with `ldapsearch` or even first just the TLS connection with `openssl s_client`?
Comment: Note `OPT_X_TLS_DEMAND` is a value for [ldap.OPT_X_TLS_REQUIRE_CERT](https://www.python-ldap.org/en/python-ldap-3.3.0/reference/ldap.html#ldap.OPT_X_TLS_REQUIRE_CERT), it is not an option as used in the example.
Answer: Okay, so basically I needed to do a couple of things. I had to add a copy of the cacert.pem file, then edit an ldap.conf file, and then I was finally able to connect to the LDAP server with TLS.
If you do not have a cacert.pem file, then you may need to make one, or ask a network admin to get one if you don't know how. See the link below for details on Using Certificates: http://www.openldap.org/faq/data/cache/185.html
Since python-ldap is based on OpenLdap, then you may need to install some dependencies as stated here: I can't install python-ldap
If ldap is installed on Ubuntu, then you should see a file structure like this:
/etc/ldap
    ldap.conf sasl2 schema slapd.d

I made a new directory in ldap called cacert, then added the cacert.pem file in there like so: /etc/ldap/cacert/cacert.pem
Then I edited ldap.conf with these changes below:
TLS_CACERT      /etc/ldap/cacert/cacert.pem
TLS_REQCERT     allow

After saving the ldap.conf, I tested the connection by entering the following in the terminal, and was finally able to connect and get results back from the server:
ldapsearch -H ldap://ldap.yourdomain.com -Z -x -D "cn=admin,dc=ldap,dc=test,dc=com" -w "P@ssWerd2LD@pP" -b "dc=ldap,dc=test,dc=com" uid=*

Then I tested my python script and was able to run it without raising any exceptions after starting TLS
I hope this helps anyone else that is having trouble connecting to LDAP with TLS on Ubuntu
Comment: I'm curious if this is the configuration you settled on. The `TLS_REQCERT allow` fixed my problem, but it concerns me. The OpenLDAP [docs](https://www.openldap.org/doc/admin21/tls.html) say this about it "there generally is no good reason to change this setting."


Flattening a nested MongoDB Document

Question: This is a follow up to
Flatten a nested object in MongoDB and rename
After using the  mentioned answer on that , some of the data types get accepted and they are flattened out while some throw  error
  can't convert undefined to object

For example:-
A document on which the data fails:-
{
  "id" : "1415304490",
  "color" : {
      "1391" : "Grey"
  },
  "name":"Random Name"
}

And the corrosponding function I am using:-
db.suggestion.find().forEach(function(doc) {
  var color;
  Object.keys(doc.color).forEach(function(key) {
    color = doc.color[key];
  });

    db.suggestion.update(
       { _id: doc._id },
       { $set:
           { color: color }
       }
    );
})

It ran on 20,000 prior documents but now failing on this document.
Also If i try to run the same code by running it on a single id , i.e
     db.suggest.find({id:"12"})

It again works fine as it should be on the same document it was failing earlier.
I also tried to create a work-around as it was running for finding one document each, i.e
db.suggestion.find().forEach(function(dest) {
  var id_temp=dest.id;

  db.suggestion.find({id:id_temp}).forEach(function(doc) {
    var color;
    Object.keys(doc.color).forEach(function(key) {
      color = doc.color[key];
    });

    db.suggestion.update(
       { _id: doc._id },
       { $set:
           { color: color }
       }
    );
  })
})

It still fails. I am quite unsure of this weird behaviour of the code.
Answer: Use $exists to make sure the color key is actually present. This is why you are getting the error.
var ops = [];

db.collection.find({ "color": { "$exists": true } }).forEach(function(doc) {
  var color;
  Object.keys(doc.color).forEach(function(key) {
    color = doc.color[key];
  });
  ops.push(
    { "updateOne": {
      "filter": { "_id": doc._id },
      "update": { "$set": { "color": color } }
    }}
  );
  if ( ops.length >= 500 ) {
    db.collection.bulkWrite(ops);
     ops = [];
  }
})

if ( ops.length > 0 ) {
  db.collection.bulkWrite(ops);
   ops = [];
}

Set up a document:
db.collection.insert(
 {
   "id" : "1415304490",
   "color" : {
       "1391" : "Grey"
   },
   "name":"Random Name"
 }
)

Run the supplied code on that and get:
db.collection.find().pretty()
{
        "_id" : ObjectId("59430fc4fc23376431161e52"),
        "id" : "1415304490",
        "color" : "Grey",
        "name" : "Random Name"
}

Comment: I agree but why should the code fail in the test document i gave above since it does have a color field
Comment: @AyushAggarwal It does not fail. I just copied your document and ran this over it "again". Same result, as it converts it perfectly. My code and not **your** code. The problem as I see it is that you are not simply applying the code you are given. The only reason I would see to not do that is that in fact your documents do not have the structure you say they do. But on the structure presented in the question it works just fine, and of course skips over anything where there is no color key to convert. That's the error in your question, and that is what is addressed.
Comment: I am making edits to your code because while running your script in robomongo , it shows an error in ops=[ ...ops ] as "unknown ." . Hence I have to change it to a workaround and make it update on by one and not in bulk as you are doing . Rest , I think there is a issue in robomongo , I'll run this in shell now. Any other mongodb GUI's that you suggest ?
Comment: @AyushAggarwal Robomongo unforntunately does not support ES6 syntax, so I just made a "very little" alteration to use `.push()` instead of the ES6 way of assigning a immutable array. I generally "recommend" using the plain old [mongo shell](https://docs.mongodb.com/manual/mongo/), because ( and especially when asking questions here ) everyone has one since it's included in every MongoDB installation. It's also the only place we can "reliably" see how data is actually structured and present, including actual data types. I suggest you get used to that when asking here. Also supports ES6.
Comment: Yeah , after your previous comment I tried using mongo shell and it worked like a charm . Thank you for all your help

Flattening a nested MongoDB Document


Understanding algebra used in $\varepsilon$-$\delta$ definition

Question: I would like to preface my questions by saying that I understand how the $\epsilon$-$\delta$ defintion of a limit works. The proof seeks to formalise the intution about a graph approaching a certain $f(x)$ value, by saying that there exists an $f(x)$ $\forall x$ between $x=a$ (limiting value) and an abritrarily chosen $x$ value in a region that is less than $\delta$ units away from $a$, but greater than $0$. Therefore, when we get as differentially close to $x=a$ as we want to and we can still find f(x) values in that region, then $\lim_{x \to a}f(x)$ truly exists.
However, the algebra always stumps me for all functions other than linear ones, where we have to limit certain factors to find an initial delta. Take for example the following:$$\mathit{Prove} \lim_{x \to 1}(x^2+9) = 10$$
This is what process I would use to prove the limit:$$\forall \epsilon \gt 0, \exists \delta : 0< |x-a|<\delta \implies |f(x)-a|<\epsilon$$ $$\therefore |x^2+9-10|<\epsilon$$ $$|x^2-1|<\epsilon$$ $$|x-1||x+1|<\epsilon$$
At this point, I get stumped. I know to assume $|x-1| \lt 1 $, but I am not sure why we are allowed to do that in the first place. So, I just assumed it was for convenience purposes since $\epsilon$ cannot be expressed in terms of $x$. But, obviously that is not an explanation of why specifically it is done.
Continuing: $$|x-1| \lt 1 \implies 0\lt x \lt 2$$ $$1\lt x+1 \lt 3 \implies |x+1|\lt3$$ $$\therefore |x-1|\cdot3 \lt \epsilon \implies |x-1|\lt \frac{\epsilon}{3}$$
So, take $\delta$ = min{1,$\frac{\epsilon}{3}$}
$$|x-1|<\delta \implies |x-1||x+1| \lt \delta |x+1| = 3 \cdot\frac{\epsilon}{3}=\epsilon$$
This last part too confuses me. I do not understand why we use the $min$ notation here and what purpose it serves in the final part where we actually prove the limit exists.
Comment: See [Epsilon-delta proof of a limit of a quadratic function](https://math.stackexchange.com/a/4354702/568204), where I've written up an answer with more details. Using $\min$ is not the only way, but it is one of the simplest ways (meaning easiest way to do things without relying on other facts, such as properties of square roots).
Comment: You may have a look at https://math.stackexchange.com/a/3659232/72031
Answer: It's not so much a question of whether we're "permitted" to assume that $|x-1| < 1$, as much as we're looking for a narrow enough $\delta$-window so that we put $f$ into the appropriate $\varepsilon$-window. The $\delta$-window can be tighter than strictly necessary, so even if we don't need to make $x$ closer to $1$ than the interval $(0, 2)$, it's OK if we do.
For instance, if $\varepsilon = 100$, it turns out that we can make $\delta = 9$. But we don't have to make it that high; $\delta = 5$ will work, as will $\delta = \pi$, as will $\delta = 1$. The only important thing is that we find a $\delta > 0$ that will work—it has to be strictly positive.
Doing so may serve, among other things, to make the reasoning easier. We might find that if we assume that $|x-1| < 1$, then we can easily conclude that a $\varepsilon/3$-window around $x = 1$ works just fine, for any $\varepsilon \leq 3$. Then, for any $\varepsilon > 3$, we know that a $1$-window around $x = 1$ will work just fine. That is why it is appropriate to write, for example, $\delta = \min(1, \varepsilon/3)$.
Comment: You only know $\varepsilon/3$ will work when $|x-1|<1$, which is why we use the minimum of the two.
Comment: Ok, I like this explanation. my only question is why will an $\frac{\epsilon}{3}$ window ONLY work for $\epsilon \lt 3$?
Comment: You don't know that it won't work for $\varepsilon > 3$, but you don't know that it will, either. And indeed, it doesn't work for large $\varepsilon$—for example, for $\varepsilon = 100$, we cannot use $\delta = 33$.
Answer: Each limit has a proper way to be approached. Normally, we assume that $0 < |x - a| < \delta_{\varepsilon}$, where $\delta_{\varepsilon}$ is a function of $\varepsilon$ which we do not know yet. So the idea is to use this relation in order to get the desired expression.
At the given case, we can proceed as follows (it is not the unique way):
\begin{align*}
|(x^{2} + 9) - 10| & = |x^{2} - 1|\\\\
& = |x - 1||x + 1|\\\\
& \leq |x - 1|(|x - 1| + 2)\\\\
& < \delta_{\varepsilon}(\delta_{\varepsilon} + 2) := \varepsilon
\end{align*}
whence we get that $\delta_{\varepsilon} = -1 + \sqrt{1 + \varepsilon} > 0$.
Hopefully this helps!
Answer: In order to prove that $\lim_{x\to a}f(x)=g$  we have to show that the value $\Delta(x)=|f(x)-g|$ is arbitrarily small provided $|x-a|$ is sufficiently small. Here we take any $\varepsilon$ and we have to show the existence of such $\delta>0$ that $\Delta(x)=|x-1||x+1|<\varepsilon$ provided $|x-1|<\delta$.
As I said, our goal is to find some $\delta$, not necessarily the best (the largest) one. We know that if some $\delta$ is good then smaller is also good. Therefore we can always restric our search for $\delta$ to the set $(0,1)$, that is we can always assume that $\delta<1$ and look what it implies. In our case this implies that $0<x<2$. This makes the second factor $|x+1|$ bounded by $3$. This makes our task a lot easier.
This technique is what I call the method of small and bounded factors. It consists in finding in each summand the factor that is small and a factor that is bounded. For example in our case $|x-1||x+1|$ the first factor is small (it can be arbitrarily small for $x$ close to $1$) and the second factor is bounded (by 3). The idea is that if $a$ is smaller than $\delta$ and $b$ is smaller than a fixed value $M$ then if we take $\delta \leq \varepsilon/M$ then $ab<\varepsilon$. Of course this doesn't give you the optimal (largest) possible value of $\delta$ but let you avoid problems with nonlinear inequalities.
To be more precise that I mean by problems with nonlinear inequalities, let's try another approach: $$\Delta(x)=|x-1|\cdot|x+1|\leq |x-1|\cdot||x-1|+2|.$$ If $|x-1|<\delta$ then $\Delta(x)<\delta(\delta+2)=\delta^2+2\delta$. This is a nonlinear inequality and we wouldn't like to be forced to solve such inequalities (in more difficult cases they can be much more complicated). Instead we factorise $\Delta(x)$ and divide our problem of finding $\delta$ into two parts. First we force $\delta$ to be just not greater than $1$. Then the second factor $\delta+2\leq 3$. Now, we see that $\Delta(x)<\delta\cdot 3$. In order to get $\Delta(x)<3$ it's sufficient to consider $\delta\leq \varepsilon/3$. So in order to make both the inequalities
$$\Delta(x)<3\delta\leq \varepsilon$$ we can take any $\delta$ that satisfies both the inequalities $\delta\leq 1$ and $\delta\leq \varepsilon/3$. Now we see that it sufficies to take the minimum of these two.

I would like to show this method in a bit more sofisticated example. We'll show that $$\lim_{x\to 1}\frac{x+5}{4-x^2}=2.$$ Let's assume that $|x-1|<\delta$ where $\delta$ is a constant that will be defined later. Let's calculate $\Delta$:
$$\Delta(x)=\left|\frac{(x-1)(2x+3)}{(2-x)(2+x)}\right| < \delta\cdot\frac{|2x+3|}{|2-x||2+x|}.$$ We managed to extract the small and the bounded factor. In order to make $\Delta$ small we need to find the upper bound of $|2x+3|$ and lower bound of $|2-x|$ and $|2+x|$.

First we assume that $\delta\leq 1$. Then $|2x+3|=|2(x-1)+5|< 2\delta+5<7$.
Now we have to be far away from $2$ and $-2$. Therefore we assume that $\delta\leq 1/2$. Then
$$|2-x|=|1-(x-1)|\geq 1-|x-1|>1-\delta\geq 1/2,\\
|2+x|=|3+(x-1)|\geq 3-|x-1|>3-\delta\geq 5/2.$$ These bounds are well seen on the axis.

Now we see that $$\Delta(x)<\delta\cdot \frac{7}{\frac 12\cdot \frac 52}=\frac{28}{5}\delta.$$ Therefore $\Delta(x)<\varepsilon$ if $\delta\leq \frac 5{28}\varepsilon$.
The last thing to do is to find what restrictions do we put on $\delta$ so that all the inequalities hold. There are:
$$\delta\leq 1,\quad \delta\leq \frac 12,\quad \delta\leq \frac 5{28}\varepsilon.$$
Therefore we can put $$\delta:=\min\left\{1,\frac 12,\frac 5{28}\varepsilon\right\} = \min\left\{\frac 12,\frac 5{28}\varepsilon\right\}.$$


How to convert a unencrypted EBS to be encrypted

Question: I have a number of older EBS volumes that are not encrypted.  In satisfying new corporate security measures, all data needs to be "encrypted at rest" so I need to convert all of the volumes to be encrypted.
What is the best way to accomplish this?
Answer: It's possible to copy an unencrypted EBS snapshot to an encrypted EBS snapshot. So the following process can be used:

Stop your EC2 instance.
Create an EBS snapshot of the volume you want to encrypt.
Copy the EBS snapshot, encrypting the copy in the process.
Create a new EBS volume from your new encrypted EBS snapshot. The new EBS volume will be encrypted.
Detach the original EBS volume and attach your new encrypted EBS volume, making sure to match the device name (/dev/xvda1, etc.)
Comment: One little gotcha. Make sure your instance supports encrypted EBS as well. It's possible you might be on an instance that doesn't. But it's just a matter of stopping the instance, then changing the type.
Comment: Just to be pedantic, you click on the unencrypted snapshot, pull down to copy, and click the encrypt button to encrypt the copy.
Comment: Wow.  Did not know that matt.  Good one.
Comment: For some reason after doing this the encrypted volume will ONLY mount as readonly...
Comment: You, man, are the MAN.
Answer: [[ This is not the right answer and not how we do things now but I'll leave this here in case anyone else finds some utility to doing it the "hard way". ]]
The following process worked well for us to convert our existing EBS volumes to be encrypted volumes.

Create a volume of the same exact size and in the same availability zone as the unencrypted volume but with encryption enabled.  If the old volume is named "XYZ", name the new volume as "New XYZ" so you don't lose track of it.  We are using the default AWS encryption keys but there are other options in the EBS docs.
Boot a temporary linux instance as the converter machine into the same availability zone as the volume.  Really any sized instance will do although EBS optimized instances may complete the migration faster.
Shutdown the instance with the current unencrypted volume.
Detach the unencrypted volume from the instance.
Attach the unencrypted volume to the converter instance.  Watch the device that the attach dialog says it is mounting as.  The first additional volume should be something like /dev/sdf.
Attach the new encrypted volume that you just created also to the converter instance.  The second additional volume will probably be /dev/sdg.
Log into the converter instance as root or as a user with sudo access.
If you look at the /proc/diststats file, at the bottom you should see something like xvdf and xvdg which correspond to the attached additional partitions.  The names may be different depending on the Linux Kernel variant/version you are using.  If there is any question, you can check the /proc/diststats file before you attach to see what partitions are added.
...
# root partition
202       1 xvda1 187267 4293 12100842 481972 52550 26972 894168 156944 0 150548 ...
# swap partion
202      16 xvdb 342 10 2810 8 5 1 48 12 0 20 20
# first attached drive, corresponds to /dev/xvdf
202      80 xvdf 86 0 688 28 0 0 0 0 0 28 28
# second attached drive, corresponds to /dev/xvdg
202      96 xvdg 86 0 688 32 0 0 0 0 0 32 32

Run the following dd command to copy from the source unencrypted volume to the destination encrypted volume.  WARNING: This command can be extremely destructive.  Take your time.  Check twice, cut once.  Have someone look over your shoulder.  These will help you from trashing your data.  Let's be careful out there!
# using a block-size of 16k (a guess), copy from input-file (if) to output-file (of)
dd bs=16k if=/dev/xvdf of=/dev/xvdg

Wait for the dd command to finish and return to the command prompt.  On our instances, a 16gb disk took ~5 minutes so you can do the math with larger.  Your mileage may vary.
Detach both the unencrypted and new encrypted volumes from the converter instance.
Attach the new encrypted volume to the instance that was using the unencrypted volume before and boot it.
When it comes up do what you need to do to validate that the system looks good.
Rename the volume from "XYZ" to be "Old XYZ".  Rename "New XYZ" to be "XYZ".  Leave around the "Old XYZ" volume in case you need to revert if there were problems.


how do I properly set up data validation?

Question: I have a DRF class which receives JSON date  and after it has been processed, create new database instances.
class CsvToDatabase(APIView):

    def post(self, request, format=None):
        data = request.data
        print()
        for _, vendor in data.items():
            v = Vendors(
                vendor_name=vendor['Vendor'],
                country=vendor['Country'],
                nda=vendor['NDA date'],

            )
            try:
                v.full_clean()
            except ValidationError as e:
                data = ({'status': str(e)})
                return Response(data, content_type='application/json')
            v.save()
        return Response({'received data': request.data,
                         'message': 'Vendors from vendors_list were successfully added to the database'})

models.py
class Vendors(models.Model):
    COUNTRY_CHOICES = tuple(countries)

    ...
    country = models.CharField(max_length=45, choices=COUNTRY_CHOICES)
    nda = models.DateField(blank=True, null=True)
    ...

tuple(countries) looks like:
(..., ('AR', 'Argentina'), ('AM', 'Armenia'), ('AW', 'Aruba'), ('AU', 'Australia'), ('AT', 'Austria'), ('AZ', 'Azerbaijan'), ('BS', 'Bahamas'), ('BH', 'Bahrain'), ('BD', 'Bangladesh'), ('BB', 'Barbados'), ('BY', 'Belarus'), ('BE', 'Belgium'), ...)

Received JSON data may or may not contain the value of the nda field. An example of an incoming data format is as follows
{
    "1": {
        "Vendor": "Firstvendortestname",
        "Country": "Belgium,",
        "NDA date": "",
        "Primary Contact Name": "Jack Jhonson",
        "Primary Contact Email": "email@example.com",
        "Secondary Contact Name": "Jack2 Jhonson",
        "Secondary Contact Email": "firstname.lastname@example.com",
        "Modules": "Module1, Module2"
    },
    "2": {
        "Vendor": "Secondvendortestname",
        "Country": "Belarus",
        "NDA date": "2019-12-24",
        "Primary Contact Name": "Sandra Bullock",
        "Primary Contact Email": "email@example.com",
        "Secondary Contact Name": "Sandra Bullock",
        "Secondary Contact Email": "email@example.com",
        "Modules": "Module1, Module2"
    }
}

What's the problem.
A first one - "Country": "Belgium," get me error
{
    "status": "{'country': [\"Value 'Belgium' is not a valid choice.\"]}"
}

But if I change to "Country": "BE," it works.
A second one - "NDA date": "", get me error
ValidationError at /api/v1/vendors/from_csv_create/
["'' value has an invalid date format. It must be in YYYY-MM-DD format."]

Request Method: POST
Request URL: http://127.0.0.1:8000/api/v1/vendors/from_csv_create/
Django Version: 2.2.9
....
.....

But at models I have null=True and blank=True. And what is strange is that the second error is printed in text format instead of json as the first. Although in the exception, I specify return Response(data, content_type='application/json').
How do I get the ability to work with an empty nda field and set up my work with?
Comment: If your view receives data that must be validated, you should be using a Serializer to validate the data. It kind of is similar to the `Form` concept from django. You will specify which fields are expected, which are required, or even define custom validation functions using `def validate_myfield(self, value):` Check out their page: https://www.django-rest-framework.org/api-guide/serializers/
Comment: @JordanKowal Yes, I know about serializer, but I think this way should work too.
Answer: For the 1st problem, it's because "Belgium" is not in your choices. Choice tuples are basically (accepted_value, human_readable_value). So when you write ("BE", "Belgium"), only "BE" will be considered as a valid choice. And it will be SHOWN as "Belgium" in your admin.
If you wanted to accept both (which might not be necessary), there are several ways to go:

Add more tuples : (("BE", "Belgium"), ("Belgium", "Belgium"))
Create a custom validator for the field

For your 2nd error "" is not valid date, it's because an empty string is not a valid value for a Date object in Django. Check out this topic, who had the same issue : DateTime Field shows invalid format error while passing empty value through form
Basically, you should either OMIT the field, or replace the "" value with None before pushing it
Comment: I seem to have obtained the desired result by changing the `clean` model method.
Comment: Thanks, its good points to problem understandings. I think I'll created new list of choices `("Belgium", "Belgium")` because I dont need both way. But the most interesting part is how to replace empty string from json to `None` value. Do you have any thoughts about that?
Comment: Yes, using .clean() is the recommended way to "intercept, control, and possibily change" incoming data.


Mann-Whitney test

Question: I am carrying out a study in which I have $3$ groups $n= 2$, $n= 5$, $n=17$. Each group has a different inner ear abnormality, following the fitting of an ear implant I have measured their speech (ordinal data) and hearing scores (continuous data) at 1 and 2 years.
The group with only $2$ patients in must be excluded from stats I believe. For the two remaining groups, most of the data are normally distributed and most variances are similar for speech (ordinal data), but not for the hearing scores (continuous data).
I wonder if a Mann-Whitney $U$-test should be used for the following reasons:

Ordinal data DV: although the data are normally distributed and most variances are similar, ordinal means that only a M-W can be done.
Scale Data DV: the data are only normally distributed with homogeneity of variance in 2/3 of the measures, therefore again M-W? Or should a $t$-test be used (but doesn't a $t$-test assume homogeneity of variance)?
Comment: I think we will need more information. Do you have measures at baseline (before implants)? What do you mean that the ordinal data are normally distributed? What do you mean that "most of the data are normally distributed"? Can you post a sample of your data, or a plot of them?
Comment: Welcome to the site, @Hannah. I took the liberty of editing your question to try to make it clearer. Please make sure it still says what you want it to.
Comment: Hello Jung and Gaetan,

Thank-you very much for your replies.

I have measured scores for speech and language using two scales known as CAP and SIR, which are rating scales from 1-7 and 1-5. These have been measured pre-implant and post implant at 12 and 24 months for both measures (CAP & SIR). I have also measured the change in scores for both CAP and SIR from pre-implant to post implant at 12 months, and then post implant at 12 months compared to 24 months.

In addition, hearing was tested before implant and at 12 and 24 months post implant.
Comment: I used Shapiro-Wilks to test normality by group, 1, 2 and 3- (IV) groups are based on their inner ear abnormality. This showed normal distribution for most of the variables, most results were > 0.05 (DV/dependant measures).

I also ran Levine's test to check for homogeneity of variance and again for most of the variables, the result was > 0.05 showing variances were similar.  However for hearing thresholds only 1 variable out of 3 had homogeneity of variance.
Comment: As Gaetan agreed and confirmed the Mann-Whitney seems appropriate for the CAP/SIR outcomes as these are ordinal.

However hearing thresholds are scale continuous scale (dB HL), they are normally distributed but homogeneity of variance is violated in 2/3 variables (DV's) measured. Therefore I wondered if again Mann-Witney would help address this issue? I wonder if a t-test could be used with a cautionary approach that not all assumptions are met.
Answer: Hannah, first you answered your own question.  Because the data is ordinal, you have to use the M-W test.  If not for this condition, you probably could have used both tests.  In many hypothesis testing situations, it is not 100% clear what test to use (you test for Normality and result is inconclusive; or you have a very large sample whereby Normality is less critical of an issue).  In such undetermined situations (this happens very often), I recommend you use both tests (t test, M-W).  You will notice that very often both tests pretty much give you very similar results in terms of statistical significance level.  So, when both tests concur on this topic your case is solved.  You can state with confidence whether you accept or reject the null hypothesis.  Much fewer cases will be anbiguous when the two tests reach opposite conclusion.  In this case, you drill down and look at your data very closely and make a conservative assumption as to given the nature of the data what test may be more appropriate.  Sometimes, it is still unclear.  In such a situation, you may decide to go with the test that gives you the more conservative result or you may decide to go with the test that reduces the type of error you are more sensitive to (Type I vs Type II error).
But, in your case because of the ordinal data, the proper test is M-W.  Just for fun, you could try using the t-test and see if the two tests would come to the same conclusion... knowing that in this case, you would have to select M-W.
Comment: (1) the Jarque-Bera test is based on the asymptotic distribution of the test statistic, and is not suitable for small or even not-so-small sample sizes  (2) it is not a good idea to use goodness of fit tests in this situation.
Comment: Glen_b, thanks for the input.  I edited my response by removing my recommendation of using the Jarques-Berra test in this situation.
Comment: Many thanks, would the Mann Whitney also be appropriate for the data I have measured which is scale. The assumptions say it is applicable to both scale and ordinal data. The issue with the t-test is that it assumes homogeneity of variances- my scale data does not meet this assumption but is normally distributed. I wonder if M-W would therefore be more robust for this outcome I am measuring?
Comment: Hannah, as long as you can rank the values, Mann Whitney will work and be meaningful.
Comment: If anyone is interested reading this, I have done some further reading and apparently a test called Kolmogorov-Smirnov Z (not to be confused with the similarly named test for normality) is much the same as Mann-Whitney but tends to have a better power than the M-W when your sample size is under 25! May be useful for anyone reading.
Comment: I thought K-S was really more appropriate for testing whether two distributions are similar or not... kind of an alternative to a Chi-Square test.
Comment: Hannah - I know of only two things that are called "Kolmogorov-Smirnov" which are the one and two sample versions of the test. The second could be used in situations where you use a Mann-Whitney but I've *never* heard it called "Z". However, to my knowledge it doesn't generally have "better power" than the Mann-Whitney against location-shift alternatives, or even stochastic ordering (which is perhaps what it sounds like you want); against some more general kinds of alternatives the two-sample Kolmogorov-Smirnov should have better power (e.g. equal means but different spreads). ...(ctd)
Comment: (ctd) ... If you can specify a specific sequence of alternatives to check against, I could compare power for you. Brief discussion of the two-sample Kolmogorov-Smirnov test is [here](http://en.wikipedia.org/wiki/Kolmogorov_Smirnov#Two-sample_Kolmogorov.E2.80.93Smirnov_test). However, the K-S isn't really suitable for ordinal data - it's meant for continuous data. If you wanted to use it with ordinal data you'd have to simulate the distribution of the test statistic under the null.
Comment: (ctd further) ... some simulations (at the samples sizes you have) seem to suggest that the two-sample K-S - while lower power than the M-W in situations the M-W is suited for - actually performs pretty well once you account for the differences in significance level due to discreteness. But you're left still with wondering why it rejected - whether due to the kind of difference you care about, or something else.
Comment: Hannah, as a matter of consistency, given that you have given me "best answer" can you also give me one helpful vote.  Thanks.

Mann-Whitney test