Interpreting a zero-inflation negative binomial model

Question: I am currently running a series of zero-inflated negative binomial models on the impact of the magnitude and direction of change in various weather parameters on a number of insect behaviours (represented as counts of the number of times the behaviour is recorded in a set period of time following exposure). My current model is examining fecundity as below. The random effects are both categorical and correspond to the day on which the behaviour was recorded and the rearing conditions of the males in the lab.
model2<-glmmTMB(No.eggs.laid~Change.6hrs*Direction.6hrs + (1|Day) + (1|Sex.ratio.line.male), family = "nbinom1",ziformula = ~Change.6hrs*Direction.6hrs + (1|Day) + (1|Sex.ratio.line.male), data = charlotte.egg)

Here is the summary output of my model:
Family: nbinom1  ( log )
Formula:
No.eggs.laid ~ Change.6hrs * Direction.6hrs + (1 | Day) + (1 |
    Sex.ratio.line.male)
Zero inflation:
~Change.6hrs * Direction.6hrs + (1 | Day) + (1 | Sex.ratio.line.male)
Data: charlotte.egg

     AIC      BIC   logLik deviance df.resid
  2980.0   3033.3  -1477.0   2954.0      430

Random effects:

Conditional model:
 Groups              Name        Variance  Std.Dev.
 Day                 (Intercept) 1.316e-02 1.147e-01
 Sex.ratio.line.male (Intercept) 8.032e-10 2.834e-05
Number of obs: 443, groups:  Day, 7; Sex.ratio.line.male, 10

Zero-inflation model:
 Groups              Name        Variance Std.Dev.
 Day                 (Intercept) 0.58920  0.7676
 Sex.ratio.line.male (Intercept) 0.01893  0.1376
Number of obs: 443, groups:  Day, 7; Sex.ratio.line.male, 10

Overdispersion parameter for nbinom1 family (): 5.42

Conditional model:
                                   Estimate Std. Error z value
(Intercept)                         2.92753    0.11709  25.003
Change.6hrs                        -0.26768    0.12984  -2.062
Direction.6hrsIncrease             -0.06377    0.13621  -0.468
Change.6hrs:Direction.6hrsIncrease  0.23991    0.15250   1.573
                                   Pr(>|z|)
(Intercept)                          <2e-16 ***
Change.6hrs                          0.0392 *
Direction.6hrsIncrease               0.6397
Change.6hrs:Direction.6hrsIncrease   0.1157
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Zero-inflation model:
                                   Estimate Std. Error z value
(Intercept)                         -0.9118     0.6278  -1.452
Change.6hrs                         -2.9058     1.3911  -2.089
Direction.6hrsIncrease              -0.8555     0.6355  -1.346
Change.6hrs:Direction.6hrsIncrease   3.3083     1.4543   2.275
                                   Pr(>|z|)
(Intercept)                          0.1464
Change.6hrs                          0.0367 *
Direction.6hrsIncrease               0.1783
Change.6hrs:Direction.6hrsIncrease   0.0229 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

I have a few questions with regards to the output of this model:

What does the zero-inflation model actually represent?
Are these p-values sufficient in interpreting the model or do further statistical tests of significance need to be completed in order to infer relationships between variables?
With this type of model how would you go about determining the significance of the random effects? With non-zero inflated models I am able to do this by using the anova() function to compare a model with and without a particular random effect however when I tried to do this only one p-value is generated. As such I am not sure if this pertains to either the conditional or the zero-inflation model.
Answer:

What does the zero-inflation model actually represent?

this is a model for the occurance of non zeros vs zeros. It can be interpreted in the same was as a logistic regression model where success means a non-zero count and you are modelling the probability of obtaining a non-zero count.

Are these p-values sufficient in interpreting the model or do further statistical tests of significance need to be completed in order to infer relationships between variables?

Try to interpret the coefficient estimates, not the p values, but yes, the p values can be interpreted, as the probability of obvserving these data, or data more extreme, if the null hypothesis is true. That is, each p value relates to a specific test of a specific null hypothesis and that is the only context in which you can interpret p values.

With this type of model how would you go about determining the significance of the random effects? With non-zero inflated models I am able to do this by using the anova() function to compare a model with and without a particular random effect however when I tried to do this only one p-value is generated. As such I am not sure if this pertains to either the conditional or the zero-inflation model.

Again, don't worry too much about p values from these tests. You have repeated measures and therefore you are accounting for this using random intercepts. It is sufficient to report the variance of these random intercepts. In your case you can note that the variance of one of these variance components in both parts of the model is small in comparison to the other. Having said that, it is good to seek a parsimoneous model, so if you have reason to believe that there should not be any correlation within either of your grouping variables for either part of the model, then you can remove the corresponding random term from the model and perform a likelihood ratio test in the same way as you do with a model without zero inflation - note that you have 2 parts to the model that include random effects: the main part and the ziformula part.
Comment: "The zero-inflation model estimates the probability of an extra zero such that a **positive contrast** indicates a higher chance of **absence** (e.g. minedno <0 means fewer absences in sites unaffected by mining); this is the opposite of the conditional model where a positive contrast indicates a higher abundance (e.g., minedno >0 means higher abundances in sites unaffected by mining." [Brooks et al., 2017] (https://journal.r-project.org/archive/2017/RJ-2017-066/RJ-2017-066.pdf)
Comment: "It can be interpreted in the same was as a logistic regression model where success means a non-zero count and you are modelling the probability of obtaining a non-zero count." -> I think this is wrong. The zero-inflation model predicts the probability of structural zeros. So we are modeling the probability of obtaining a (structural) zero, not a non-zero as erroneously stated.


Fazer ordenação por Reflection com GetProperty

Question: Tenho uma classe genérica com um método que precisa ordenar um objeto genérico.
Porém, dar erro dizendo que não reconhece o método Reflection GetProperty, pois lambda não consegue traduzir.
Como posso fazer essa lógica de ordenação genérica?
public IEnumerable<TEntity> GetAll()
{
    var obj = _repository.GetAll()
        .OrderByDescending(x => x.GetType().GetProperty(typeof(TEntity).Name + "Id"));

    return obj.Pagination();
}

Segue a imagem do erro:

---- Edit ----
Refatorando (rsrs) o meu problema de forma resumida é que eu quero que um Serviço, seja qual for, quando quiser fazer uma consulta, mesmo que não passe nenhum parâmetro, o programa já limitar com paginação, boas práticas me entendem né! Abaixo alguns pontos para entenderem e no final coloquei o link do pastebin para visualizarem as classes ExtensionsQueries.cs, Service.cs e ClienteService.cs. Coloquei apenas parte das classes, espero que seja suficiente.

Tenho uma classe genérica para meus Serviços herdarem onde contém os métodos comuns a todos os Serviços, como GetAll(), GetBy(Expression<Func<TEntity, bool>> filter), Insert(TEntity obj), etc...;
TEntity é uma Entidade Mapeada do Banco de Dados genérica que esta classe Service irá usar;

Até aqui está tudo funcionando perfeito, agora comecei a implementar uma forma de paginação que seja genérica tbm, então comecei a modificar algumas classes para adptar e foi aí que tive esse problema, comecei a fazer o seguinte:

Criei um Extension Method de IQuerable para fazer Paginação do retorno de uma consulta;
O problema é que o Skip só funciona se houver uma Ordenação antes;
Já que vou ter que implementar uma Ordenação, estou querendo parametrizar o GetAll() e os demais que precisarão e receber algum atributo para ser usado na ordenação, tipo GetAll(obj paramsToOrder), mas se não passar (não será obrigatório) ordenar pela ChavePrimaria, que no meu caso é o Nome da Classe com o Id no final (que foi onde comecei a implementar como viram no exemplo que postei);
Nem sei ainda como passar esses parâmetros, pois a idéia é passar N atributos para ordenar e cada um ter seu tipo de ordenação ASC ou DESC. Só que antes de chegar nessa parte já tive esse problema. T_T

Link: https://pastebin.com/ZiyjiurL
Comment: @ramaral Sempre tem uma moda nos abusos. Embora ainda seja a exceção o maior abuso, reflexão está se tornando forte agora. Parece que está virando moda usar reflexão por usar, sem nenhuma necessidade e benefício. A reflexão com um literal sempre é um "erro", ela só é útil quando não se sabe o que deseja usar. Outro dia dei uma resposta sobre isso e deu uma confusão, porque quando a pessoa decide que vai usar o mecanismo, ninguém tira da cabeça :) Espero que neste caso seja diferente. Como L2E usa reflexão fica bem complicado funcionar reflexão em cima de reflexão.
Comment: Se fosse o caso de ser útil usar, se não soubesse mesmo o nome do campo, teria que fazer a reflexão fora para pegar o nome e aí usar isto na *query*. Ou então usar SQL mesmo :)
Comment: Qual a razão da necessidade de usar *reflection*? Por que não usar `OrderByDescending(x => x.Id)`?
Comment: @bigown a minha questão surge exactamente por esse facto: ser conhecido não só o tipo como o nome do campo. Já agora, me passe o link da resposta que refere.
Comment: possivelmente você chamou esse método, informando o `TEntity` como uma `string`, aí ele tentou chamar `GetProperty(typeof(string))`. Posta a declaração da classe onde está esse método também
Answer: Já tentou fazer assim?
_repository.GetAll().OrderByDescending(x => Convert.ToInt32(x.GetType().GetProperty("Id").GetValue(x, null).ToString())).ToList();
Aqui ele ordena os objetos pelo Id de forma decrescente. Caso quiser crescente alterar para OrderBy


Digits of irrationals

Question: I've been studying floating point arithmetic and I've read somewhere that numbers with infinitely many decimal digits without recursion are irrational.
But since we can't know all the digits of such a number then how did we come to the conclusion that its digits have no recursion? Does it have anything to do with formulae used to compute the $n$-th digit of a number?
(This is a question simply out of curiosity.)
Comment: Maybe reword the "read somewhere" part as "reals whose decimals are neither terminating nor eventually repeating are irrational."
Comment: You can know easily if the digits are specified algorithmically, as in $\alpha =\sum 10^{-n^2}$.  Otherwise, your instincts are reasonable...this isn't a good way to show that a number is irrational.
Comment: You usually prove that a number is irrational by contradiction, assuming that it is rational and getting a contradiction. See http://www.homeschoolmath.net/teaching/proof_square_root_2_irrational.php
Comment: The question should specify which number you are talking about. Could be mathematical constants, experimental data or results of numerical computations. These situations are very different.
Answer: The simpler (and ancient) way to know if a number $a$ is irrational is to explicitly show that it cannot be expressed as a quotient $\frac{n}{m}$ of two integers $n,m$.
But there are numbers, as the number $\pi+e$,  for which we don't know if they are rational or irrational.
Answer: Floating-point (as well as fixed-point) arithmetic is just unable to represent irrationals and most rationals.
Actually, the floating-point numbers are essentially integers in a finite range, with a movable point, and can't have more than 16 (significant) decimal digits.
Comment: More generally I would say that any computable number can be represented by a finite program which outputs its digits. Alas, we won't know if it's irrational in most cases
Answer:
I've read somewhere that numbers with infinitely many decimal digits
  without recursion are irrational.

This property can indeed be used to determine whether some numbers are irrational. For instance, the prime constant, defined by
$$
\rho =\sum _{{p}}{\frac  {1}{2^{p}}}=\sum _{{n=1}}^{\infty }{\frac  {\chi _{{{\mathbb  {P}}}}(n)}{2^{n}}},
$$
where the sum goes over all primes $p$ and ${\displaystyle \chi _{\mathbb {P} }}$ is the characteristic function of the primes, has a $1$ as its $k^{\rm th}$ digit if $k$ is prime, and $0$ otherwise. It can be shown using an argument similar to yours that $\rho$ is irrational. Another example is the constant whose every $2^{k^{\rm th}}$ digit is $1$ and every other is $0$:
$$
0.0101000100000001\ldots,
$$
which is also irrational.

But since we can't know all the digits of such a number then how did
  we come to the conclusion that its digits have no recursion?

As you rightly observe, in other cases irrationality is proved by other means. For instance, the irrationality of $\sqrt 2$ is proved by assuming first that it is rational, i.e., equal to some $\frac ab$, where $a\neq b$ and $a,b>1$ are positive integers, and coming to a contradiction (for a nice presentation see here). A slightly more difficult proof using different techniques is used to show that the number denoted by $\zeta(3)$, where $\zeta$ is the Riemann zeta function, is irrational.
Answer: The proof is that if the number had a series of recurring digits, then it would be rational.  Suppose the number is a.bbbrrrr  where bbb is the leader digits and rrrr is a recuring patten.
Then this number is $a + \frac b{10^B} + \frac r{10^{B+R}-10^B}$, where b is the digit string bbb, and B is the number of places it occupies, and r, R are the recurring digits, and the period length.
Since this number is rational in every case, there is no recurring pattern that is not rational.

Digits of irrationals


Android on Intellij showing error: is not an activity

Question: I have just imported my android project into intellij from eclipse. I am new to intellij.
I have gotten everything working and set up the run configuration for Android. When I run it, it gives me the edit configuration screen.
When I say launch the default activity, it tells me that it can't find the default activity, which is odd. When I code the activity to launch straight away, it tells me that that's not an activity subclass or alias. I double-checked and the class is certainly an activity.
When I ignore it and run anyway, it blows up as soon as it hits the phone.
Can you help me understand/fix these errors, please?
Answer: In the Intellij project structure dialog, select the module that your Activity is residing in.
Then under the sources tab, add the root folder of your sources as sources for this module by clicking the 'Mark as Sources' link or button.  The sources root folder should be mark blue.
The module can now find your sources and launch the activity.
Comment: Excellent, thank you very much! I missed the fact that the sources were not correctly marked.
Answer: My solution was to delete and recreate the run configuration.
Answer: In my case, none of these solutions worked. Then I started digging.
As I remember, my java files had to be placed in src/main/java/com.something.blabla. Then I realized that the directory has changed to  src/main/java/com/something/blabla.
So I created a new com.something.blabla folder under ../java/ and moved my java files to that folder. I removed the com folder, and it worked perfectly.


How to create a view with no data

Question: I have a content type with a number of fields, one of which is state (pending/active/complete). Users can also be assigned to a piece of content. Eg Frank is linked to content titled "Blah" with state "active".
I want to create a view that shows users who ARE NOT linked to any content with the state field "active".
So far, I figured I can show a list of users and then using relationships, I pull up and reference the content.
It is easy enough to show users linked to content that has the state field "active", but for the life of me, I cannot figure out how to show users who ARE NOT linked to content with state "active".
Any ideas?
Comment: This can be accomplished using views. Try the video series http://dev.nodeone.se/en/taming-the-beast-learn-views-with-nodeone to get a handle on how to use it :)
Comment: Checked out the Beast video series and I still can't see how to do it. The issue is that I am trying to dynamically exclude or include data, based on other data included or excluded. Eg If X has Y, then don't show any other X's, even if they show Z'ds.
Answer: Thank you all.
In the end I added a new field to the user as Jimajamma suggested, which counts the number of state (active) nodes that user has. I used Rules to add and subtract from the number to keep it current. Views then filters, only reporting on users with 0 state (active) nodes.
Answer: From what I'm able to understand, you want to be able to see users that are not linked to that piece of content type. Thus, could you possibly just create a view that uses a contextual filter for users NOT of that content type?
Comment: Unfortunately this won't work. It is the same content type, with a field marked state, so all users will be using that content type, but not all will have that state as active.
Answer: I'd say go the other way round:
Create a view of Users and add a relation to the nodes.
Then, in your filter, set your state field to Pending or Inactive.  You can select multiple values IIRC.
Comment: Nice try - getting closer :) Unfortunately I can't go "NULL", as the field often has other values in there.

I need the filter to exclude any user who has "active" on ANY node. ie I want it to look at more than 1 node and if any of them have "Active" as a state, then it should exclude ALL of the nodes from the result (or exclude the user of all of those nodes from the result)
Comment: I hate to say this but I think this might be a case of a poor architectural decision on your behalf.  This is straightforwards and shouldn't be so hard! The only other way I can think of is using a hook_views_query_alter().
Comment: I'll take that on the chin, but I'm pretty sure the design is OK :)
I simply have a single field named state that can contain the values Pending, Active and Complete. So it will never return null in a view as, right from node creation it is given the value "Pending".
It looks like I will need to add another field specificially to track "Active" (Y/N), but to me, this seems less architectually sound. I'll also look at the various workflow modules to see if they can be used.
Comment: Hum I guess I misunderstood your field's purpose, I was under the impression it allowed other kinds of values (hence the remark about the architecture).  Updated my answer!
Answer: One way I can see this being done easily is to have another field linked somehow to your user that is incremented when the user is assigned or otherwise has active content.  In D6, this could be as simple as having an integer field in a content profile.  Using permissions you could set it up so the user can't change or edit it and then whenever a user is assigned a piece of active content, this gets incremented.  When that content is no longer active, or assigned to another user, it gets decremented. This logic could easily be put inside nodeapi().  Then you could use a view to look for users with this field being zero.  You could also look for overworked users, too :)
Comment: I came to the same conclusion - Another field would do the trick - It does seem the clumsy option though :)
Comment: The issue here is that I don't see that there is a direct way with the joins that views uses to do what you want.  You could put a multiple value field in the user side of the house that links to the content and then all would be good in the world but given you are doing it in the opposite way, I don't see how to do it directly.  You could create a view that returns all the users with content assigned to them, shove those uids in an array and then create another array of all your users uids and then use array_diff() to find out who isn't in there...all pretty clumsy.
Comment: another way would be to create an array of all your uids, and then use views to grab the uids of content owners and then cycle through them, unsetting the corresponding user in the first array, and then at the end, you'd have an array of users without content.
Comment: Thank you all.

In the end I added a new field to the user as Jimajamma suggested, which counts the number of state (active) nodes that user has. I used Rules to add and subtract from the number to keep it current. Views then filters, only reporting on users with 0 state (active) nodes.
Answer: If I can understand correctly, you just want a view of users not connected to "active" content right?
Here's what I would do:

Create a User view
Add an entity reference relationship for your user reference field (I'm assuming this is how you are linking users)
Add two filters (normal, not contextual) for field_state; first one is "is not one of: active" which will display users linked to pending/complete content and second one is "is empty" which will display users who are not connected to anything
Create a new filter group (click the arrow next to "add" and select "rearrange") and change the operator to OR
Put those two filters you created in this OR group, and you should be good to go.

This worked for me on a new test bed I made just now, so it SHOULD work for you too.  But I am not sure how complicated your site currently is so your mileage may vary.  Let me know if it works.
Comment: It does work as you indicate, but what still doesn't produce the result I want:
Comment: Sorry Craig, I guess I haven't fully grasped what you need.  And come to think of it, I think I basically duplicated Alex's suggestion, whoops!

May I ask where this list of users (is that right?) will be displayed?  Is it a page? Block on a piece of content?

How to create a view with no data


Como converter CSV para XLSX com python?

Question: Como eu faço para converter um arquivo .csv gerado pelo python para .xlsx?
Estou com dois problemas:

Um deles é que eu não consegui entender como fazer esta conversão
O segundo é que mesmo passando o comando
crawl <nome> -o <nome>.csv -s CSV_DELIMITER=";"

no arquivo scrapy, na hora que eu abro o arquivo gerado direto pelo Excel, o mesmo não formata as colunas, deixando separado por vírgulas.

Meu código:
# -*- coding: utf-8 -*-
import scrapy
from scrapy import *
from Aranhas.items import ImprensaNacional
import csv
from Aranhas.settings import *
from Aranhas.pipelines import *

class imprensa_Nacional(scrapy.Spider):
    name = 'imprensa_Nacional'
    start_urls = ['http://www.imprensanacional.gov.br/leiturajornal?data=11-09-2018&secao=dou3']

    custom_settings = {
        'FEED_EXPORT_FIELDS': [
            'Titulo', 'Tipo', 'Pregao', 'UASG', 'DOU', 'DataPublicacao', 'EdicaoDou',
            'SecaoDou','Pagina', 'Orgao','Pregoeiro', 'Url', 'Descricao'
        ],
        'FEED_FORMAT': 'csv'
    }

    def parse(self, response):
        url_base = 'http://www.imprensanacional.gov.br/'
        script = response.xpath('//*[@class="span8 hierarchy-wrapper"]//*[contains(text(),"AVISO DE LICITA")]')
        for urls in script:
            links = urls.xpath('.//@href').extract_first()
            link_completo = url_base + str(links)
            yield Request(url=link_completo, callback=self.parseAviso)

    def parseAviso(self, response):
        conteudo = response.xpath('//*[@class="journal-content-article"]')
        for info in conteudo:
            titulo = info.xpath('.//*[@class="identifica"]/text()').extract_first().strip()
            pregao = info.xpath(u'.//*[@class="identifica"]/text()[contains(.,"N\xba")]').extract_first().split()[3].strip()
            uasg = info.xpath(u'.//*[@class="identifica"]/text()[contains(.,"N\xba")]').extract_first().split()[6].strip()
            tipo = info.xpath('.//*[@class="identifica"]/text()[contains(.,"AVISO")]').extract_first().strip()
            pregoeiro = info.xpath('.//*[@class="assina"]/text()').extract_first().strip()
            descricao = info.xpath('.//*[@class="dou-paragraph"]/text()').extract_first().strip()
            dou = info.xpath('.//*[@class="dou-paragraph"]/text()[contains(.,"(")]').extract_first().strip()
            orgao = info.xpath('.//*[@class="orgao-dou-data"]/text()').extract_first().strip()
            data_publicacao = info.xpath('.//*[@class="publicado-dou-data"]/text()').extract_first().strip()
            edicao_dou = info.xpath('.//*[@class="edicao-dou-data"]/text()').extract_first().strip()
            secao = info.xpath('.//*[@class="secao-dou"]/text()').extract_first().strip()
            pagina = info.xpath('.//*[@class="secao-dou-data"]/text()').extract_first().strip()

            item = ImprensaNacional()
            item.set_all()

            item['Titulo'] = titulo.encode('iso-8859-1')
            item['Tipo'] = tipo.encode('iso-8859-1')
            item['Pregao'] = pregao.encode('iso-8859-1')
            item['UASG'] = uasg.encode('iso-8859-1')
            item['DOU'] = dou.encode('iso-8859-1')
            item['DataPublicacao'] = data_publicacao.encode('iso-8859-1')
            item['EdicaoDou'] = edicao_dou.encode('iso-8859-1')
            item['SecaoDou'] = secao.encode('iso-8859-1')
            item['Pagina'] = pagina.encode('iso-8859-1')
            item['Orgao'] = orgao.encode('iso-8859-1')
            item['Pregoeiro'] = pregoeiro.encode('iso-8859-1')
            item['Url'] = response.url.encode('iso-8859-1')
            item['Descricao'] = descricao.encode('iso-8859-1')

            yield item

Answer: Outro legal também (além do Pandas) é o rows (instalável com pip install rows, mas depois vai pedir a instalação manual de alguns outros pacotes, também com pip, par funcionar).
É uma biblioteca em Python para tratamento de tabelas, que além de facilitar a leitura e processamento de vários formatos de arquivo e dados em páginas na Web, disponibiliza, uma vez instalado, o comando rows no terminal, que pode fazer a conversão deseada de forma tão simples quanto digitar rows convert arquivo.csv arquivo.xlsx.
Answer: Veja pandas, onde você pode ler um csv e usar a função ExcelWriter e to_excel para fazer as escritas.
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html
Outro pacote que pode te ajudar é o Openpyxl:
https://openpyxl.readthedocs.io/en/stable/


Can't open new port in iptables

Question: Today I'm facing a strange problem.
Some days ago I configured Iptables. I opened port 80,1000 and some others port.
Today I need to open port 25 for mail system. I have added the rule in configuration file of iptables,  but for some reason the port stay closed.
I also tried to restart the system, but still port 25 remains closed. I also tried to disable iptables... and port 25 remains closed. I have no idea of what's happening...
I'm on Centos 7
EDIT: Of course, the port I opened some days ago are still opened
EDIT 2: Here's the result of the command iptables -L -n -v
    Chain INPUT (policy DROP 1173 packets, 75761 bytes)
 pkts bytes target     prot opt in     out     source               destination
19620 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19620 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19620 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
19633 3555K f2b-default  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
15679 3181K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
 2849  353K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:20
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:21
   61  3524 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:50492
  140  7600 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
    2   104 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995
    2   104 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993
  183  9889 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10000
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spts:1024:65535 dpts:1024:65535 state ESTABLISHED
18543  597K ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 35337 packets, 4328K bytes)
 pkts bytes target     prot opt in     out     source               destination
 1824  797K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spts:1024:65535 dpts:1024:65535 state RELATED,ESTABLISHED

Chain f2b-default (11 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       220.127.116.11        0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       184.108.40.206        0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       188.8.131.52      0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       22.214.171.124       0.0.0.0/0            reject-with icmp-port-unreachable
 216K   39M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Comment: Post the output of 'iptables -L -n'
Comment: Actually, post the output of `iptables -L -n -v` - @ArulSelvan, `-v` includes vital information like restrictions on interface, without which rules cannot be meaningfully examined.
Comment: @MadHatter I am not the OP. I am responding to your `-v` comment above. I allow port 80,443 from anywhere to my host. Here is the output of `iptables -L -n |egrep '80|443'` where you can see the rule. I agree adding `-v` is good but not necessary in this case.

`# iptables -L -n |egrep '80|443'`
`ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80`
`ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443`
Comment: @ArulSelvan I apologise for my confusion, but you are wrong about not needing `-v`.  Consider the rule `iptables -I INPUT -i tun+ -p tcp --dport 80 -j ACCEPT`.  Without `-v`, this will look identical to your blanket permit on inbound HTTP traffic; but it won't work like that in practice (it permits only inbound HTTP traffic from logical interfaces normally associated with established routed OpenVPN connections).  When analysing possible rule failures, **`-v` is always needed**.
Comment: I edited the first post adding the output of iptables -L -n -v
Answer: The evidence is strong that your rule is taking effect; see the non-zero packet counts on the rule:
 61  3524 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25

So if you find you still can't connect, it would be useful to see the evidence.  It'd also be worth checking that your MTA is listening on the external address.  Could you add into your question the output of netstat -apn|grep -w 25 (needs to run as root)?
If it looks like
tcp        0      0 :::25                       :::*                        LISTEN      1813/sendmail

then I'm wrong, and all is well on that front.  But if it looks like
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1814/sendmail

then your MTA is configured to listen only on the loopback address (127.0.0.1:25), and you need to fix that.
Answer: Just a little question - is there anything what is listening on port 25? Is your mailserver running? Port is closed if firewall block it OR when nothing is running on that port.

Can't open new port in iptables


How to correctly implement multithreading using a Tkinter GUI?

Question: I am trying to implement Multithreading while using a GUI in tkinter. I came along this solution, but i am failing to implement it.
So basically i need to know:
How do i need to alter my Code to make the Progressbar interact nice and smoothly, i.e. not going in unresponsive mode when the GUI loses focus?
This is my code boiled down:
from tkinter import *
import queue
import threading
from tkinter.ttk import *

class ThreadedTask(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

# Gui class
class MyGui(Frame):
    def __init__(self, parent):
        super().__init__(parent)
        self.parent = parent
        self.queue = queue.Queue()
        self.init_ui()

    # define a button to fulfill long task
    def init_ui(self):
        self.frame = Frame(self, relief=RAISED, borderwidth=1)
        self.frame.grid(row=0, column=0, columnspan=1, sticky='ns')
        self.status_frame = Frame(self, relief=RAISED, borderwidth=1, height=20)
        self.status_frame.grid(row=1, column=0, columnspan=3, sticky='nesw')
        self.status_frame.grid_configure(padx=3, pady=3)
        self.button = Button(self.frame, text='do Stuff', command=self.do_stuff)
        self.button.grid(padx=3, pady=3)
        self.progress = Progressbar(self.status_frame, orient="horizontal", length=80, mode="determinate")
        self.progress.grid(row=1, column=0, pady=3, sticky='nesw')
        self.grid()

    # start ThreadedTask here
    def do_stuff(self):
        ThreadedTask(self.queue).start()
        self.queue_process(DoStuffClass(ui=self))

    def queue_process(self, process, retry_time=10):
        self.master.after(retry_time, process)

    def update_status(self):
        self.parent.update_idletasks()

class DoStuffClass:
    def __init__(self, ui=None):
        self.ui = ui
        self.long_task()

    def long_task(self):
        # do stuff here and update the progressbar from MyGui
        self.ui.progress['maximum'] = 10000
        # some lengthy task ...
        for i in range(10000):
            print(i)
            self.ui.progress.step()
            self.ui.parent.update_idletasks()

# main
root = Tk()
root.geometry("150x80+50+50")
MyGui(root)
root.mainloop()
root.quit()

I think right now my problem is the wrong implementation of queues and Threading, i.e. self.queue_process(DoStuffClass(ui=self))... the behaviour is the same as if i wouldn't use a queue and Multithread at all. The progressbar works, as long as it stays "in focus", which means me not clicking anything else on the desktop. As soon as i'm clicking elsewhere on the desktop and the GUI loses focus, the GUI goes in "unresponsive" mode and the progress bar does not update anymore. Also, sometimes Tcl closes the wrong Thread, which makes the whole program crash.
Comment: Your question is very broad and there are too many sub-questions (some of which make little sense — i.e. there's no such thing a sub-thread). The main thread is the first code the Python interpreter starts running. You can implement `ThreadedTask` class anywhere you want. Threads aren't spawned from it, instances of it **_are_** separate threads. The basic thing to keep in mind is that only one thread (usually the main one) should be updating the GUI (via `after()`). Other threads need to put stuff in `Queue`s and the main thread should check them and update the GUI depending on what it finds.
Comment: You could also have a look at the .after() method (in Tkinter) that will allow you to run a function in Tkinter without pausing the GUI. This will also remove the need to use the threading module.
Comment: You boiled-down your code too much. When I try to run it: `AttributeError: 'MyGui' object has no attribute 'preview_button'` from line `self.preview_button.grid(padx=3, pady=3)`. BTW, the main thread is the one that is executed first by the Python interpreter.
Comment: You may find [Freezing/Hanging tkinter Gui in waiting for the thread to complete](https://stackoverflow.com/questions/53696888/freezing-hanging-tkinter-gui-in-waiting-for-the-thread-to-complete) helpful.
Comment: @martineau Hello Martineau, **i've updated my Code**. I tested it and it works until the line `self.queue_process(DoStuffClass(ui=self))` which is the part i am uncertain with. However, you do see a progressbar moving while doing the lengthy task, and when u change focus of the minigui, it will go in unresponsive mode (and for my original program -- sometimes crash).
Answer: So after a couple of tries i figured out what to do:
from tkinter import *
import queue
import threading
from tkinter.ttk import *

class ThreadedTask(object):
    def __init__(self, parent):
        self.parent = parent
        self.queue = queue.Queue()
        self.gui = MyGui(parent, self.queue)
        self.work_queue()

    def work_queue(self):
        """ Check every 100 ms if there is something new in the queue. """
        try:
            self.parent.after(200, self.work_queue)
            print('working queue with task {}...'.format(self.queue.get_nowait()))
        except queue.Empty:
            pass

# Gui class
class MyGui(Frame):
    def __init__(self, parent, queue):
        super().__init__(parent)
        self.parent = parent
        self.queue = queue
        self.init_ui()

    # define a button to fulfill long task
    def init_ui(self):
        self.frame = Frame(self, relief=RAISED, borderwidth=1)
        self.frame.grid(row=0, column=0, columnspan=1, sticky='ns')
        self.status_frame = Frame(self, relief=RAISED, borderwidth=1, height=20)
        self.status_frame.grid(row=1, column=0, columnspan=3, sticky='nesw')
        self.status_frame.grid_configure(padx=3, pady=3)
        self.button = Button(self.frame, text='do Stuff', command=self.init_button_loop)
        self.button.grid(padx=3, pady=3)
        self.progress = Progressbar(self.status_frame, orient="horizontal", length=80, mode="determinate")
        self.progress.grid(row=1, column=0, pady=3, sticky='nesw')
        self.grid()

    def start_thread(self, function_name, queue):
        t = threading.Thread(target=function_name, args=(queue,))
        # close thread automatically after finishing task
        t.setDaemon(True)
        t.start()

    # execute button push by spawning a new thread
    def init_button_loop(self):
        self.start_thread(self.exec_button_loop, self.queue)

    # execute long task
    def exec_button_loop(self, queue):
        self.progress['maximum'] = 10000
        for i in range(10000):
            # update progressbar
            queue.put(self.progress.step())

# main
root = Tk()
root.geometry("150x80+50+50")
client = ThreadedTask(root)
root.mainloop()

The Difficulty was to figure out how to interact with queues and threads, while pressing buttons in the gui.
Basically my fault was to declare the queue in the wrong class and the wrong use of the after function while not knowing where to start the threads.
The new implementation follows the principle of filling the queue up in a new thread, which is spawned when pressing the gui button. The queue is periodically checked from the main thread whether theres something to do. This prevents unresponsiveness due to a safe communication between gui and mainthread.


A group theory question relating equality of cosets

Question: If $x,y$ belongs to a group $G$ and $x=y$, then can i directly conclude that $xH=yH$ for any subgroup  $H$ of $G$?
Comment: Sure...it's true element by element.
Answer: Of course, if $x=y$ then $\forall_{h\in H} \; xh = yh$. Thus, every element of $xH$ is contained in $yH$ and vice versa.
Cheers!


Display a different login screen to the user based on IP Address. Any ideas why this is not working?

Question: We have an app that allows users from both within the organization or outside.
However, if the user is from our organization, we would like to authenticate the user's account using our Active Directory.
If the user is coming from outside, we would like the user to register to obtain an acount.
We felt that the easiest way to handle this is to track their ip addresses from the subnet of their networks.
This seems simple enough but when I tried running the code below, it is continously showing the screen that is intended for outside users.
What am I doing wrong?
'//relevant markup:

<asp:Panel id="pnlLoggedOut" runat="server" Visible="True">
    Login or Register
</asp:Panel>
<asp:Panel id="Panel2" runat="server" Visible="False">
    Login
</asp:Panel>
<asp:Panel id="pnlLoggedIn" runat="server" Visible="False">
    <asp:Label ID="userLB" runat="server" ForeColor="#CC0000"></asp:Label>[Logout]
</asp:Panel>

'//relevant codeBehind:
Protected Sub Page_Load(ByVal sender As Object, ByVal e As EventArgs) Handles Me.Load

    'Create a variable to hold your IP address.

    Dim ipAddress As String = ""

    'Retrieve the user's ip address from Page_Load():

    ipAddress = Request.ServerVariables("REMOTE_ADDR")

    'Then compare the retrieved IP address to the subnet for our network which is usually " 255.255.255.0".

    If ipAddress.Contains(" 255.255.255.0") Then
        pnlLoggedOut.Visible = False
        Panel2.Visible = True
    Else
        pnlLoggedOut.Visible = True
        Panel2.Visible = False
    End If

    'Response.Write(Session("UserRole"))
    If Session("UserName") Is Nothing Or Session("UserName") = "" Then
        pnlLoggedOut.Visible = True
        pnlLoggedIn.Visible = False
    Else
        pnlLoggedIn.Visible = True
        pnlLoggedOut.Visible = False
        userLB.Text = "You are logged in as " & Session("FullName")
        userLB.ForeColor = System.Drawing.Color.DarkOrange
    End If
End Sub

Comment: Have you checked in the debugger, which value is stored in the `REMOTE_ADDR` header when running? I guess "255.255.255.0" is a customized sample value, isn't it?
Comment: @Markus, That's actually our subnet address.

However, when I tried debugging, REMOTE_ADDR is getting the value of 127.0.0.1.

That's not going to help.
Comment: 127.0.0.1 is the localhost address (that is local by definition). 255.255.255.0 usually is a subnet mask that can be used to determine whether two ip addresses belong to the same subnet.
Comment: @Markus, Right, do you have any idea what changes are neeed in my code then?

I thought that using the subnet mask is the best way to determine source of incomng IP Address (inside or outside).
Answer: IP Address is not a reliable way to catch this so here is an alternative:-
I think it is a better option is to configure IIS to enable integrated security then you can do something like this:
If System.Web.HttpContext.Current.User.Identity.IsAuthenticated Then
    'Internal
Else
    'External
End If

Answer: In order to solve this based on the IP addresses, you need to retrieve the IP addresses of the server. You can achieve this with this function (see this question and its answer for details):
Dim host = Dns.GetHostEntry(Dns.GetHostName())
Dim ipAddresses = From x In host.AddressList
                  Where x.AddressFamily = System.Net.Sockets.AddressFamily.InterNetwork

Then you need to check whether the incoming IP address is part of the same subnet as any of the server's IP addresses. This article on MSDN can show you some useful functions, in particular the IsInSameSubnet extension method. Converted to VB.NET the following code should work:
Imports System.Net

Public Module IPAddressExtensions

    <System.Runtime.CompilerServices.Extension> _
    Public Function IsInSameSubnet(address2 As IPAddress, address As IPAddress, _
                                   subnetMask As IPAddress) As Boolean
        Dim network1 As IPAddress = address.GetNetworkAddress(subnetMask)
        Dim network2 As IPAddress = address2.GetNetworkAddress(subnetMask)
        Return network1.Equals(network2)
    End Function

    <System.Runtime.CompilerServices.Extension> _
    Public Function GetNetworkAddress(address As IPAddress, subnetMask As IPAddress) As IPAddress
        Dim ipAdressBytes As Byte() = address.GetAddressBytes()
        Dim subnetMaskBytes As Byte() = subnetMask.GetAddressBytes()

        If ipAdressBytes.Length <> subnetMaskBytes.Length Then
            Throw New ArgumentException("Lengths of IP address and subnet mask do not match.")
        End If

        Dim broadcastAddress As Byte() = New Byte(ipAdressBytes.Length - 1) {}
        For i As Integer = 0 To broadcastAddress.Length - 1
            broadcastAddress(i) = CByte(ipAdressBytes(i) And (subnetMaskBytes(i)))
        Next
       Return New IPAddress(broadcastAddress)
    End Function

End Module

In order to check whether an IPAddress is internal or external, do the following. Note that interalIP denotes an IP address from the subnet that you want to define as the internal subnet. If your server is located in a DMZ that has its own subnet, you need to configure the IP address instead of getting it dynamically as shown above.
Dim internalIP = new IPAddress("192.168.1.1") ' Taken from configuration or determined as above
Dim externalIP = new IPAddress(Request.ServerVariables("REMOTE_ADDR"))
Dim subnetMask = new IPAddress("255.255.255.0")
If internalIP.IsSameSubnet(externalIP, subnet) Then
    pnlLoggedOut.Visible = False
    Panel2.Visible = True
Else
    pnlLoggedOut.Visible = True
    Panel2.Visible = False
End If

Comment: @MattWilko, your solution is very cool but I can't use it because the app will be out on dmz. It is outside of our domain.

Markus, I getting errors on the function, something like:

`etNetworkAddress is not a member of the System.Net.IpAddress`

Thanks to you guys for your assistance.
Comment: @ChidiOkeh: The errors were due to another extension method that is needed. After you add it, it should work. However, if it will run in the DMZ, be sure to check whether the IP addresses of the server (in the DMZ) are relevant for checking the subnet (an alternative would be to configure the relevant subnet). Also, authentication against AD might be difficult from the DMZ.
Comment: about authenticating against AD, you are right. However, we are dumping employee details from AD to SQL Server and authenticating against this SQL Server table instead. Works just fine as we have done it before. Still getting the error though even after adding the additional method. `GetNetworkAddress is not a member of the System.Net.IpAddress`
Comment: @ChidiOkeh: the error says that the extension method cannot be found. This might be related to namespace imports in the file; also the type and method should be public. I've pasted the complete class from the MSDN page into a file in a test project and it worked.
Comment: how many namespaces did you import?

I have already imported the two that I am aware:

`Imports System.Net
Imports System.Net.IPAddress` and they are all public.

Thanks again.
Comment: @ChidiOkeh: it is impotant that the Namespace that contains the class with the extension methods is imported. Either create a new class in your project and copy-paste the complete class from the MSDN article over it or convert the extension methods to normal ones by removing the `this` in front of the 1st parameter and supply the parameter explicitly. Hope this helps.
Comment: thanks for all your help.

Maybe, the reason I am running into the error that I showed above is because I am converting the codes to vb.net since that's the flavor I am using for my app.

I tried pretty much everything you recommended but no love.

Thanks for all your help though.Much appreciated.
Comment: @ChidiOkeh: sorry, I missed that. I've converted the code in the answer. Extension methods are defined in another way in VB.NET. Hope this solves the issues.
Comment: super!

I put the module in App_Code folder to resolve this new error:

`module can only occur at file or namespace levels`

Then I put the other code in page_load() event.

Now, no more errors but just one last question - I promise!

How do I use it then to compare internal vs external IP Addresses?

Thanks alot Markus for your resolve and patience.
Comment: @ChidiOkeh: I've updated the sample. I didn't test it, so I hope it works.


How to get bulk reports via Twilio API?

Question: I'm going to get reports over call/message on Twilio via API. They have a limitation of 1000 reports on a single page. The requested GET query is something like this https://api.twilio.com/2010-04-01/Accounts/{Account_Sid}/Calls.json?PageSize=1000. I want to get them and show them to the user through a pie chart in Asp.net Core MVC. It returns a next_page_uri  property for each request that you make and you'll get the reports for the next 1000 if you make a request to that. I tried it with Postman and it works fine, but I failed to get it worked via HttpClientFactory in Asp.net Core MVC. I think I should iterate through a collection  of next_page_uri and make a GET request for every one, but It's not a collection.
try
            {
                var request = new HttpRequestMessage(HttpMethod.Get, $"https://api.twilio.com/2010-04-01/Accounts/{_configuration.GetSection("TwilioAuth:ACCOUNT_SID").Value}/Messages.json?DateSent={report.Date}&PageSize=1000");
                var client = _httpClientFactory.CreateClient("MyClient");
                HttpResponseMessage response = await client.SendAsync(request);
                if (response.StatusCode == HttpStatusCode.OK)
                {
                    var jsonResponse = await response.Content.ReadAsStringAsync();
                    model = JsonConvert.DeserializeObject<ReportMessageModel>(jsonResponse);
                    foreach (var message in model.messages)
                    {
                        if (message.direction == "outbound-api")
                        {
                            outboundMessages.Add(message.status);
                            ViewBag.messageCount = outboundMessages;
                            if (message.status == "delivered")
                            {
                                delivered.Add(message.status);
                                ViewBag.delivered = delivered.Count;
                            }
                            else if (message.status == "undelivered")
                            {
                                undelivered.Add(message.status);
                                ViewBag.undelivered = undelivered.Count;
                            }
                        }
                    }
                }
                if (model.next_page_uri != null)
                {
                    var newRequest = new HttpRequestMessage(HttpMethod.Get, $"https://api.twilio.com{model.next_page_uri}");
                    HttpResponseMessage newResponse = await client.SendAsync(newRequest);
                    if (newResponse.StatusCode == HttpStatusCode.OK)
                    {
                        var jsonResponse = await newResponse.Content.ReadAsStringAsync();
                        model = JsonConvert.DeserializeObject<ReportMessageModel>(jsonResponse);
                        foreach (var message in model.messages)
                        {
                            if (message.direction == "outbound-api")
                            {
                                outboundMessages.Add(message.status);
                                ViewBag.messageCount = outboundMessages;
                                if (message.status == "delivered")
                                {
                                    delivered.Add(message.status);
                                    ViewBag.delivered = delivered.Count;
                                }
                                else if (message.status == "undelivered")
                                {
                                    undelivered.Add(message.status);
                                    ViewBag.undelivered = undelivered.Count;
                                }
                            }
                        }
                    }

                }

            }

Answer: Since you are using C#, why not use the C# SDK and let it handle the MessageResource paging for you?
// Find your Account SID and Auth Token at twilio.com/console
// and set the environment variables. See http://twil.io/secure
string accountSid = Environment.GetEnvironmentVariable("TWILIO_ACCOUNT_SID");
string authToken = Environment.GetEnvironmentVariable("TWILIO_AUTH_TOKEN");

TwilioClient.Init(accountSid, authToken);

var messages = MessageResource.Read(dateSent: {report.Date});

    foreach(var record in messages)
    {
       Console.WriteLine(record.Sid);
    }

An alternative, is to use the Usage Records:
 var records = RecordResource.Read(
        category: RecordResource.CategoryEnum.Sms,
        startDate: {report.Date},
        endDate: {report.Date}
    );

The above will return data quickly but only be a total count which won't allow for complex graphics.
Also, as an aside, I think you will find the MessageResource.Read approach not efficient (slow for end-user reporting). A more efficient approach is to use Twilio's webhooks to collect the data in your own database. Then you can query your database however and how often you prefer. Twilio describes this in the REST API Best Practices document:

If you are performing a large amount of GET requests, consider
implementing webhooks aka StatusCallBack requests for the resource
endpoint(s) your account is utilizing. The information contained in
the responses posted to your servers will often remove the need to
perform any future polling GET requests.
If you are frequently fetching the same data from Twilio, we recommend
moving the data from Twilio to your own servers. Once data has been
successfully moved, delete data stored on Twilio servers if you no
longer need it. Not only will this will reduce costs, this is also a
generally recommended business practice for privacy, security, and
compliance.

How to get bulk reports via Twilio API?


When to use taxonomy vs content type?

Question: I have a project that allows people to add 'skills' and 'resources' to a system. Initially I was using a taxonomy field for the objects themselves because it seemed to make it easy to look up other 'skills' and 'resources' that had already been added to the system.
However, I am beginning to wonder whether this is the best plan.
The taxonomy system allows for a 'library' of objects to be created which can be shared but this isn't the right way to use taxonomy which should be for 'meta-information'.
I could have a preexisting taxonomy structure of 'stuff' like ebay's categorie and allow people to add their own stuff to this library. People would add 'skills' and 'resources' as their own content types with additional fields- title, body, image, and then tag them within this taxnonomy structure. However, I was attracted to the idea of keeping it really simple; e.g. a drill is only a drill not a bosch, 12v battery powered handheld drill with hammer setting and masonry bit :)
If I wanted to have a 'drill' categories with a pool of 'drills' with more specific drill information what would be the best way to do that?
Sorry there are a few questions here, but I'm hoping to put together question an exciting distribution from this project, so any thoughts will help and hopefully go to something worthwhile.
Answer: Content type defines kind of information, different content types means different structure and different nature of content. Taxonomy is for categorisation. They can be interchangeable to some extent, but it's really up to how you want your data defined.
If all nodes we are talking about perform the same function in the system and are supposed to have the same fields, keep them categorised by taxonomy. If their function should be different, make content types - it does not mean you need to give up taxonomy then (but you may if it's no longer needed).
Comment: I could have skills and resources as their own content types, and then asking for tags on these too, but my fear is that there will be a lot of disparate information entered into the content-type, but then again, I could lock this down quite a lot with specific fields. Thanks for the input.
Comment: @AndrewWelch I can't include future information in my answer. I could give you guidelines how to decide, but only you can apply them to your exact situation.

When to use taxonomy vs content type?


Using xmlhttp to load multifile from server

Question: I use xmlhttp to load text file from server. I can only load 1 file, how can I load multifile? here is the code :
      function load_controll_unit_enable_disable_default()
      {
        var xmlhttp=new XMLHttpRequest();
        xmlhttp.onreadystatechange=function()
        {
            if (xmlhttp.readyState==4 && xmlhttp.status==200)
            {
                var arr = [];
                arr = xmlhttp.responseText.split("\n");
            }
        }
        xmlhttp.open("GET","file.txt",true);

        xmlhttp.send();
       }

Answer: HTTP is limited to receiving only one response per one request. If you want to get more in one swoop try serializing using JSON. For example request /files which returns a JSON array:
[ { 'file': 'file.txt', 'data': ... }, { 'file': '...', 'data': ... }, ... ]

JSON.parse it and read each filedata. Note that JSON does not support binary data, only UTF-8, so for binary you'll probably have to armor it (base64 btoa and atob, etc.).


How would you write something like x-y+z=1 as a vector?

Question: Specifically, the question is asking me to prove whether or not the collection of vectors [x,y,z] in R3 forms a subspace of R3 or not. The numbers for one question of this were z=2x and y = 3x. This is easy enough to understand because I can write a vector all in terms of x and see if the properties of a subspace hold for that generalized vector.
But how might I write x-y+z=1 as a vector? The best I can do is write one variable as two of the others, but that doesn't get me anywhere. Is there something I'm missing?
Sorry if this is a stupid question, any help is appreciated.
Comment: What are the axoims that a set must satisfy in order to be a subspace?
Comment: I like to first translate to matrix $[1~-1~1]^t[x~y~z] = 1$, then a can set $a=[1~-1~1]$ and $v = [x~y~z]$, so in a pure vector sense $a\cdot v = 1$
Comment: As @Yanko says, the question asked is "Do the vectors $(x, y, z)$ satisfying $x - y + z = 1$ form a vector space $V$"? To do this, you should look at things like, suppose, $(x_1, y_1, z_1) \in V$ and $(x_2, y_2, z_2) \in V$, is $(x_1 + x_2, y_1 + y_2, z_1 + z_2) \in V$, using the definition of $V$. Similarly the other axioms of vector spaces should also be checked.
Comment: It can be easily seen that the plane $x-y+z=1$ is not a subspace of $\Bbb R^3$ because it doesn't contain the zero vector, $\vec 0$
Answer: Writing the set of all $x,y,z$ such that $x+y+z=1$ as a vector makes no sense mathematically. But I believe you mean the following:
$$\{[x,y,z]\in\mathbb{R}^3: x+y+z=1\} = \{ [x,y,1-x-y] : x,y,z\in\mathbb{R}\}$$
Now we can write $[x,y,1-x-y]$ as $[x,0,-x]+[0,y,-y]+[0,0,1]$. In other words you have that
$$\{[x,y,z]\in\mathbb{R}^3: x+y+z=1\} = \{[0,0,1]+x\cdot [1,0,-1] + y\cdot [0,1,-1] : x,y\in\mathbb{R}\}$$
This is a simple form of the given set which I believe is what you meant by "writing $x+y+z=1$ as a vector". From this expression it might be easier to deduce that the set is not a subspace of $\mathbb{R}$.
Comment: Thank you very much, this helps a lot!


VMWare Server MUI Login Help

Question: I'm very new to VMWare server, or at least, installing it and I'm having issues with the MUI.  I have read many places that generally speaking, the root user is the only one that can access the MUI from the get-go.  However, I cannot seem to get this to work.  I've tried everything I can think of.  I'm running Jaunty Ubuntu and the most recent VMware server version.
Any thoughts?
Thanks,
Chris
Comment: Can you GET to the MUI, and are just unable to authenticate, or is the problem with getting to the MUI in the first place?
Comment: I can get to it, view the login screen and attempt to log in.  I enter root/password and get a permission denied error
Answer: Well you can run it outside the root account... I created a user vmware and used that as the user and specified that during the installation...  Anyway, is the root account enabled and password set?  By default Ubuntu disables the root account.  You can the enable root account by issueing this command:
<code>sudo passwd root
</code>
Enter your sudo password, if needed. Then enter password you want to use for root.  However keeping root enabled and using the root account is not the best practice...
Comment: No beans.  I'm still having issues authenticating.  From what I recalled, it never asked me what account I wanted to use, is there a way to set that retroactively?
Comment: It uses the built in OS accounts.  By default, only the root user has access to login.
Comment: @Adam That's what I thought.  Any idea why it isn't working?
Answer: I am running 1.0.9 on CentOS 5.3.  In each case I have the VMware guests set with user ownership of root and group ownership of vmware and permissions set as 775. (user root and group vmware rwx and other as r-x).  Ideally I would run not as root but think I need root access to be able to set the NIC into promiscuous mode.  I haven't played with that yet.
I then have several OS users as members of the vmware group.  When they log in using their OS account information, they can see and manage all guests.
If I want a guest to be private, I set its user and group ownership to be that private user.
Comment: You must be root to set promiscuous mode, correct.  However, that is only needed on VMWare Server startup, not for the MUI or individual VMs.

VMWare Server MUI Login Help


Brauer characters: independence over $\overline{\mathbb{Q}}$ implies independence over $\mathbb{C}$

Question: The proof of theorem in title has already been sketched in a question on MathStack (here); I tried to write the detailed proof, and I want to check it.
Let $\phi_1,\cdots,\phi_n$ be all the Brauer characters of a finite group $G$ (so they have domain $G_{p'}=\{g\in G: p\nmid o(g)\}$, and values in $\mathcal{O}\subseteq \mathbb{C}$, the ring of algebraic integers; $p$ is the characteristic of field $F$ over which the inequivalent irreducible $F$-representations of $G$ are defined with above Brauer characters.
Claim: Suppose $\phi_1,\cdots,\phi_n$ are independent over $\overline{\mathbb{Q}}$; then they are independent over $\mathbb{C}$.
Proof: (1) First if $|G_{p'}|=m$, then  $m$ should be $\leq n$, otherwise $\phi_i$'s can-not be independent.
(2) Let $G_{p'}=\{g_1,g_2,\cdots,g_m\}$. For each $g_i$ we associate a row-vector
$$[\phi_1(g_i),\phi_2(g_i),\cdots,\phi_n(g_i)]\in \mathbb{\overline{Q}}^n.$$
We get $m$ vectors $\{v_{g_1}, v_{g_2},\cdots,v_{g_m}\}$ in $\overline{\mathbb{Q}}^n$, and $m\leq n$.
(3) The $\overline{\mathbb{Q}}$-independence of $\phi_i$'s is equivalent to say that these $m$ rows in $\overline{\mathbb{Q}}^n$ are independent over $\overline{\mathbb{Q}}.$ Extend this set to a basis of $\overline{\mathbb{Q}}^n$:
$$\{v_{g_1}, v_{g_2},\cdots,v_{g_m}, w_{m+1},\cdots, w_n\}.$$
(4) The matrix $P$ formed by these $n$ vectors as rows of a matrix $M_n(\overline{\mathbb{Q}})$ will be invertible.
(5) Hence the matrix $P$ as an element of $M_n(\mathbb{C})$ will be invertible.
(6) Hence $\phi_1,\cdots,\phi_n$ as functions into $\mathbb{C}$ are $\mathbb{C}$-independent.
Is this proof correct?
Answer: Your proof looks fine to me. One thing I'll point out is that this really has nothing to do with the fact that you are dealing with Brauer characters.
You have some vectors $v_1, v_2, ... , v_n$ in a finite dimensional vector space $V$ (here class functions on $G_{p'}$) over one field $k$ (in this case $\bar{\mathbb{Q}}$) that are linear independent and you want to make sure they remain linearly independent when you extend scalars to a field extension $K$ ($\mathbb{C}$ in this case). More precisely you want to know that $v_1\otimes 1, v_2 \otimes 1, ... $ are linearly independent inside $V \otimes K$. Your proof works fine in this level of generality.
I'll note that in fact $V$ need not be finite dimensional for this result to hold, your proof would need to be modified a bit for the infinite dimensional case (as you complete to a basis and show a certain matrix is invertible) but it's not that hard to fix.


Slide effect with JQuery

Question: I'm not very good with Javascript/Jquery, and I don't understand something.
I display a full size image when the user arrives for the first time on the homepage. If the user scrolls, the image disappears slowly and the homepage appears.
To hide the image, I use the jquery function .hide(). It works correctly, but I want another animation because the default isn't very elegant.
To do that, it seems I need to include core/jquery.ui, but even if I include it in my dependencies for my library, I get the following error:

TypeError: r.easing[this.easing] is not a function

And if I include core/jquery.ui.effects.slide, I get this error:

TypeError: g[i] is undefined

Here my code:
my_module.libraries.yml
home:
  js:
    js/home.js: {}
  dependencies:
    - core/jquery
    - core/drupal
    - core/drupalSettings
    - core/jquery.once
    - core/jquery.cookie
    - core/jquery.ui.effects.slide

js/home.js
(function ($, Drupal, drupalSettings) {

  'use strict';

  Drupal.behaviors.pre_home = {
    attach: function (context, settings) {
    $(window).scroll(function () {
      var popup = $('#pre-home');
      console.log("Hide"); // Display correctly when I scroll
      popup.hide("slide", {direction: "top"}, 1000); // Doesn't work in this case, work I remove all parameters but not with the effect I want.
        });
    },
  };
})(jQuery, Drupal, drupalSettings);

I don't know what I'm doing wrong.
Comment: Is this file spelled right? `my_module.librairies.yml`?
Comment: Syntax error in my post, sorry. My file has the good name and it's correctly loaded.
Comment: I fact, I follow this post: https://stackoverflow.com/questions/40112027/jquery-hide-animation-slide-right Jquery-ui include a hide function with different effects.
Comment: @ClaireD turns out that `{direction: "top"}` is invalid. It needs to be `{direction: "up"}`. and using `- core/jquery.ui.effects.slide` of course.
Comment: Had you used `{direction: "right"}` you would've figured that out pretty quickly.
Answer: A big thank to @No Sssweat  for the answer. My methodology was good, I done an error in the definition of the direction.
So, you need to include the core/jquery.ui.effects.slide in your libraries.yml file and follow this documentation http://api.jqueryui.com/slide-effect/.

Slide effect with JQuery


Default to and select first item in Tkinter listbox

Question: I want to automatically select the first item in the listbox. By selecting the first item, I don't mean simply defaulting to the first item or setting focus on it. I've already achieved that by doing self.listbox.select_set(0). I want the default item also selected. In other words, when I run my code below, I want print(value) to print the value of the default selection. If Asia is chosen from the optionmenu, Japan should automatically print to the console. If Africa, Nigeria should print and Germany for Europe.
Any suggestion on how I can achieve this? Thanks.
from tkinter import *
from tkinter import ttk
import tkinter.messagebox

class App:
    def __init__(self):
        self.master = Tk()
        self.di = {'Asia': ['Japan', 'China', 'Malaysia', 'India', 'Korea',
                            'Vietnam', 'Laos', 'Thailand', 'Singapore',
                            'Indonesia', 'Taiwan'],
                     'Europe': ['Germany', 'France', 'Switzerland'],
                     'Africa': ['Nigeria', 'Kenya', 'Ethiopia', 'Ghana',
                                'Congo', 'Senegal', 'Guinea', 'Mali', 'Cameroun',
                                'Benin', 'Tanzania', 'South Africa', 'Zimbabwe']}
        self.variable_a = StringVar()
        self.frame_optionmenu = ttk.Frame(self.master)
        self.frame_optionmenu.pack()
        options = sorted(self.di.keys())
        self.optionmenu = ttk.OptionMenu(self.frame_optionmenu, self.variable_a, options[0], *options)

        self.variable_a.set('Asia')
        self.optionmenu.pack()
        self.btn = ttk.Button(self.master, text="Submit", width=8, command=self.submit)
        self.btn.pack()

        self.frame_listbox = ttk.Frame(self.master)

        self.frame_listbox.pack(side=RIGHT, fill=Y)
        self.scrollbar = Scrollbar(self.frame_listbox )
        self.scrollbar.pack(side=RIGHT, fill=Y)
        self.listbox = Listbox(self.frame_listbox, selectmode=SINGLE, yscrollcommand=self.scrollbar.set)
        self.variable_a.trace('w', self.updateoptions)

        self.scrollbar.config(command=self.listbox.yview)
        self.listbox.pack()

        #Populate listbox
        for each in self.di[self.variable_a.get()]:
            self.listbox.insert(END, each)
            self.listbox.select_set(0) #This only sets focus on the first item.
        self.listbox.bind("<<ListboxSelect>>", self.OnSelect)

        self.master.mainloop()

    def updateoptions(self, *args):
        #countries = self.di[self.variable_a.get()]
        self.listbox.delete(0, 'end')
        for each in self.di[self.variable_a.get()]:
            self.listbox.insert(END, each)
            self.listbox.select_set(0) #This only sets focus on the first item.
        self.listbox.pack()

    def submit(self, *args):
        var = self.variable_a.get()
        if messagebox.askokcancel("Selection", "Confirm selection: " + var):
            print(var)

    def OnSelect(self, event):
        widget = event.widget
        value = widget.get(widget.curselection()[0])
        print(value)

App()

Running Python 3.4.1
Answer: The simplest solution is to generate the <<ListboxSelect>> event at the same time that you change the selection:
def updateoptions(self, *args):
    ...
    self.listbox.select_set(0) #This only sets focus on the first item.
    self.listbox.event_generate("<<ListboxSelect>>")
    ...

Answer: # add before .mainloop()
self.listbox.selection_set( first = 0 )

EDIT#1 2014-08-21 13:50 [UTC+0000]
Tkinter.Listbox()-es have quite a complex MVC-Model-Part behaviour. Thus its Controller-Part .methods() are bit more complex subject to handle.
The Listbox() default select-mode allows only a single item to be selected, but the
select-mode argument supports four settings: SINGLE, BROWSE, MULTIPLE, and EXTENDED ( the default is BROWSE ). Of these, the first two are single selection modes, and the last two allow multiple items to be selected.
These modes vary in subtle ways.
For instance, BROWSE is like SINGLE, but it also allows the selection to be dragged.
Clicking an item in MULTIPLE mode toggles its state without affecting other selected items.
And the EXTENDED mode allows for multiple selections and works like the Windows file explorer GUI—you select one item with a simple click, multiple items with a Ctrl-click combination, and ranges of items with Shift-click-s.
Multiple selections can be programmed with code of this sort:
listbox = Listbox( aWindow, bg = 'white', font = ( 'courier', fontsz ) )
listbox.config( selectmode = EXTENDED )                         # see above
listbox.bind( '<Double-1>', ( lambda event: onDoubleClick() ) ) # a lambda-wrapped CallBackHANDLER()
# onDoubleClick: get messages selected in listbox               # not listed here
selections = listbox.curselection()                             # tuple of digit-string(s), aTupleOfSTRINGs, where digit-string(s) range from { 0, 1, .., N-1 }
selections = [ int( x ) + 1 for x in selections ]               # transform string(s) to shifted int(s), make 'em { 1, 2, .., N }

When multiple selections are enabled, the .curselection() method returns a list of digit strings giving the relative numbers of the items selected, or it returns an empty tuple if none is selected.
Beware this method always returns a tuple of digit strings, even in
single selection mode.
Thus symmetrically the Listbox().selection_set() method has to feature-rich so as to be able to configure all possible states for the value of <aSelectionSET>.
Q.E.D. above in the initial post.
Comment: Thank you for your answer. When posting code, please also try to explain it, so that the OP can understand the given code.
Comment: What does all of the comments related to the modes have to do with selecting the first item? This answer seems overly complex for such a simple problem.
Comment: @user3666197 Thanks for your explanation. I don't wish to use `MULTIPLE` mode at the moment. Anyway, I added `self.listbox.selection_set( first = 0 )` before `self.master.mainloop()` as you suggested but there is no change in behavior. You can copy the modified code from http://ideone.com/8VzU3p and test it. Any further thoughts? Thanks.
Comment: @BryanOakley Any idea how I might solve this? Thanks.


"ConnectionResetError" What should I do?

Question: # -*- coding: UTF-8 -*-

import urllib.request
import re
import os

os.system("cls")

url=input("Url Link : ")

if(url[0:8]=="https://"):
    url=url[:4]+url[5:]

if(url[0:7]!="http://"):
    url="http://"+url
try :
    try :
        value=urllib.request.urlopen(url,timeout=60).read().decode('cp949')
    except UnicodeDecodeError :
        value=urllib.request.urlopen(url,timeout=60).read().decode('UTF8')
    par='<title>(.+?)</title>'

    result=re.findall(par,value)
    print(result)

except ConnectionResetError as e:
    print(e)

TimeoutError is disappeared. But ConnectionResetError appear. What is this Error? Is it server problem? So it can't solve with me?
Comment: it would be nice if you could add the full traceback
Comment: the homepage is [here](http://jakjeon.icems.kr/main.do)
Comment: @VigneshKalai What is the traceback? how can i do?
Comment: see this so [question](http://stackoverflow.com/questions/3702675/how-to-print-the-full-traceback-without-halting-the-program) there they have expalined about traceback
Comment: It prints where the error has occurred and what the error was it will be easy to decode if we know that
Comment: @VigneshKalai The Error is in "result=re.findall(par,value)" The Error name is " NameError: name 'value' is not defined
Comment: That is because the value variable was not initialized that is the value variable did not get value in both try and expect both throwed error
Comment: we have to do debug step by step ok try just typing all imports and this line  `value=urllib.request.urlopen(```your url````,timeout=60).read().decode('cp949')` and don't take any user input
Comment: @VigneshKalai "SyntaxError: invalid syntxt" Error
Comment: @VigneshKalai [Picture](http://imgur.com/1NZAh3a)
Comment: url should be in `"` that is `"http:/www.google.com"`
Comment: @VigneshKalai I don't know this Error T.T [Picture](http://imgur.com/tSfFKO0)
Comment: did you `import traceback`
Comment: @VigneshKalai Oops! Sorry Here is [Picture](http://imgur.com/6QV59Qd)
Comment: remove result and all just add imports and run `value=urllib.request.urlopen(```your url````,timeout=60).read().decode('cp949')` what error are you getting
Comment: @VigneshKalai You means this? [Picture](http://imgur.com/u2xxqCz)
Comment: OMG don't give "google" give you school url and you are right
Comment: @VigneshKalai HaHa Sorry Sorry mistake [Picture](http://imgur.com/LNoHyHw)  The last sentence means "In this connection it was forcibly closed by the remote host"
Comment: could you use other modules then urllib.requests
Comment: @VigneshKalai Nope T.T I can use only urllib.requests since I'm beginner
Comment: No idea of the error Googled it then to was not able to find it .All the best mate :D
Comment: @VigneshKalai I cant understand what you saying.sorry
Comment: There are people great then me you will eventually find the answer there is no need to be sorry I gained certain knowledge because of this question like wise you would have
Answer: 포기하지 마세요! Don't give up!
Some website require specific HTTP Header, in this case, User-agent. So you need to set this header in your request.
Change your request like this (17 - 20 line of your code)
# Make request object
request = urllib.request.Request(url, headers={"User-agent": "Python urllib test"})

# Open url using request object
response = urllib.request.urlopen(request, timeout=60)

# read response
data = response.read()

# decode your value
try:
    value = data.decode('CP949')
except UnicodeDecodeError:
    value = data.decode('UTF-8')

You can change "Python urllib test" to anything you want. Almost every servers use User-agent for statistical purposes.
Last, consider using appropritate whitespaces, blank lines, comments to make your code more readable. It will be good for you.

More reading:

HTTP/1.1: Header Field Definitions - to understand what is User-agent header.
21.6. urllib.request — Extensible library for opening URLs — Python 3.4.3 documentation - Always read documentation. Link to urllib.request.Request section.
Comment: Thank you This problem is complete. Are you Korean?
Comment: "python urllib test" is not affected ?
Comment: Yes. You can change it to any string which is not empty. Yes. I'm korean :)
Comment: 감사합니다 ㅎㅎ. 저 글은 본인이 영작하신건가요? ㄷㄷ 영어실력이 대단하시네요.
Comment: 한글로 질문해도 되나요? 어떻게 사이트의 헤더가 User-agent인지 알아내는 방법을 알고 싶습니다.
Comment: Connecting to http://jakjeon.icems.kr/main.do from browser is accepted, while urllib is not, so it means that site checks something in http request. From that point, You can check what is required. And, NO, YOU MUST USE ENGLISH.
Comment: Ok.. It only exist "User-agent" in headers?


tc traffic limiting/shaping/throttling/policing in linux ipv6 issues

Question: I managed to discover an interesting issue: I'm not getting proper limiting with ipv4 traffic, but instead it works perfectly with ipv6.
This doesn't make much sense to me since I read this article: http://lartc.org/howto/lartc.adv-filter.ipv6.html
I've simplified down to the following bash script to do my throttling:
#!/bin/bash
set -x

DOWNLINK=375
UPLINK=375
DEV=eth0

# clean existing down- and uplink qdiscs
tc qdisc del dev $DEV root
tc qdisc del dev $DEV ingress

###### uplink
tc qdisc add dev $DEV root handle 1: htb default 20

# Add the classes
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbps ceil ${UPLINK}kbps
tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbps ceil ${UPLINK}kbps prio 1
tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbps ceil ${UPLINK}kbps prio 2

# both get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10

# TOS Minimum Delay gets higher prio (interactive traffic, ie ssh)
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
      match ip tos 0x10 0xff  flowid 1:10

# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
    match ip protocol 1 0xff flowid 1:10

# Add ack packets to the prio queue
tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8 0x10 0xff at 33 \
   flowid 1:10

########## downlink #############
# attach ingress policer:
tc qdisc add dev $DEV handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbps burst 10k drop flowid :1

Here's the original script I was working with and I couldn't get it to work right:
    #!/bin/bash
    set -x
#start over
tc qdisc del dev eth0 handle ffff: ingress
tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1: htb default 30

tc class add dev eth0 parent 1: classid 1:1 htb rate 375kbps ceil 375kbps

#This is for ports 22, 80, 873 and 3400
tc class add dev eth0 parent 1:1 classid 1:22 htb rate 50kbps
tc class add dev eth0 parent 1:1 classid 1:80 htb rate 1kbps ceil 50kbps
tc class add dev eth0 parent 1:1 classid 1:873 htb rate 1kbps ceil 50kbps
tc class add dev eth0 parent 1:1 classid 1:3400 htb rate 25kbps ceil 50kbps

# My default
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 25kbps ceil 25kbps

tc qdisc add dev eth0 parent 1:22 handle 22: sfq perturb 10
tc qdisc add dev eth0 parent 1:80 handle 80: sfq perturb 10
tc qdisc add dev eth0 parent 1:873 handle 873: sfq perturb 10
tc qdisc add dev eth0 parent 1:3400 handle 3400: sfq perturb 10
tc qdisc add dev eth0 parent 1:30 handle 40: sfq perturb 10

# Port 80, 873, 3400 and 22 traffic
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:80
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 873 0xffff flowid 1:873
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 3400 0xffff flowid 1:3400
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 22 0xffff flowid 1:22

tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 375kbps burst 10k drop flowid :1

Interestingly enough, I discovered that they both work correctly FOR IPv6 CONNECTIONS ONLY IPv4 continues to be UNTHROTTLED.
Interestingly enough, IPv4 traffic shows up in tc -s -d qdisc show dev eth0 it just doesn't shape or rate limit it. (Except ingress traffic, that rate limits perfectly)
I just want to make that clear, as most people have the opposite problem.
Linux myserver.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

I am running dual stack with both IPv4 and IPv6, and require both to be active.
Any suggestions or help on how to get this working in at least ipv4 (prefer both)?
Answer: tc -s -d -r filter show dev eth0 may say more about what filters match.


A melodic 2nd above a simple minor triad has a unique feel, and a required duration. Explanations?

Question: Examples
"Long" (first sung note) in Carpenters 'Superstar'.  B mel above A minor
"Far" in 'Yesterday' (McCartney).  E mel above D minor
The effect is lost over a min7 (why?). Or without extended duration.
Do we hear it as 11/5 or 22/5 over the root?
As in lyric "news" from 'All the young dudes' (Bowie), it is often high note sung in the section, and of special impact.
Comments?
Comment: Hi Garett, welcome to the forum. Your question is not very clear. You use terms that are not common (B Mel over…, 11/5). Furthermore it is not clear what exactly you are asking, apart from generic thoughts or personal opinions on this harmonic pattern.
Comment: What is a "melodic" second? Never heard of it. You can have minor seconds and major seconds. Minor seconds can be dimished like a G sharp followed by an A flat; major seconds can be augmented like a G natural followed by an A sharp.
Answer: The technical term is non-chord tone (NCT.)
In a nutshell, super simplified, the idea is in homophonic music the principle tones of a melody are derived from tones of the chord progression, and those principle tones can be embellished with non-chord tones.
The standard way of defining non-chord tones is by how they are melodically approached and resolved to a chord tone. Some times non-chord tones are described as accented or unaccented depending on how they are placed rhythmically either on of off of strong beats.
The particular examples you give involve the tone a whole step above the chord root resolving down to the root. Standard classification doesn't take into account which chord tone is embellished. Also, standard classification doesn't take into account the duration a non-chord tone. You could hold it as long as you want.
In All The Young Dudes...

...the F# is a chord tone to chord D, then the line leaps up to B which is a non-chord tone to the Am chord, then it steps down to resolve to the tone A which is a chord tone. That kind of melodic movement, leaps to a NCT and resolving down by step is called an appoggiatura.
In Yesterday...

...there is sort of combination of two NCT types, an auxiliary tone for the E F E movement above an A7 chord, and the three note descending line F E D over the Dm chord. Your concern is the E above a Dm chord. The F and D are chord tones and the E fills the space between them moving in steps, that filling in the space of the thirds of chords is called passing motion or a passing tone.
Of course neither the auxiliary tone or the passing tone movements are complete, the first ends a bit short, and the second starts early. But the motions are close enough that I believe the NCT terms I use make sense if you really need to give them  labels.

...unique feel...special impact. Comments?

Non-chord tones are considered dissonant. Consonance and dissonance are fundamental harmonic elements related to stability and tension. The ebb and flow of dissonance resolving to consonance creates a strong forward impulse in music. Also, the tension of dissonance is often treated as a way to expression emotional intensity. So, for example, if you wanted to convey "heartbreak", you might try place some appoggiatura on minor chords. If that type of thing is overdone, or too obvious, I can sound like "Mickey Mousing." Much of the time a musician uses NCTs in a more intuitive way.
The NCT concept is from "classical" theory and is very much concerned with harmony first. I think you can also look at these two examples from a more pop music frame of mind that isn't so concerned with harmony and melody tones that don't "fit the chords."
In All The Young Dudes, the melody repeats the basic line of D E F# A, but the second iteration adds the B to A variation. Variation like that keeps the line alive and developing. You could say the melodic variation aspect is more important that the NCT aspect.
In both songs the NCT works so that rhythmically the point where the melody sings the chord root is moved off of beat one. That is a kind of syncopation. The melody gets to the root tone "late." From a rhythmic perspective making the melody not hit the chord root on beat one is a sort of avoidance of an ending and that works to keep the line moving forward. In this rhythmic sense, these NCTs are a way to add syncopation and de-emphasize beat one.
Answer: Doesn't have to be a long note. Doesn't have to be over plain minor rather than m7. It's generally considered a 9th rather than 2nd, due mainly to the root and 3rd being way too close in pitch. Yes, it can be 'm add9', but the m7 fits well too, making it 'm9'.
Another great example is 'Laura', which consists mainly of 'add9' in the melody.


is that possible in java? (code to execute) if (condition)

Question: Please note that I am not a java guru. I might not use the right terminology, I learn java inside RFT on the fly. What is described below works exactly as stated.

In ruby we can code like
(code to execute) if (condition)

I want to do the same so my RFT (Rational Functional Tester) code is easy to read. I am going to call my custom functions in a way that looks like
        findANDclick(new String[]{"id", "menuButton"});
        findANDclick(new String[]{"src", ".*homeicon_calendar.*"});
        findANDclick(new String[]{"src", ".*cycle_templates.*"});

But the whole RFT script needs to finish and don't execute any other code in case of any of the findANDclick functions 'failed'. The function searches for an object with in html page and if it doesn't find any it throws new Exception via
throw new Exception("findANDclick: the object was not found");

so the instance of findANDclick ONLY throws an error so the next findANDclick is executed. But it makes no sense to continue as look for next object if the previous was not found and clicked on.
I was thinking that I can have a variable continue set to true and in case the exception is thrown the findANDclick will update it to false. Then I can do something like
    if (continue) { findANDclick(new String[]{"id", "menuButton"});}
    if (continue) { findANDclick(new String[]{"src", ".*homeicon_calendar.*"}); }
    if (continue) { findANDclick(new String[]{"src", ".*cycle_templates.*"});   }

it would be great if I can do something like
        { findANDclick(new String[]{"id", "menuButton"}); } if (continue)
        { findANDclick(new String[]{"src", ".*homeicon_calendar.*"}); } if (continue)
        { findANDclick(new String[]{"src", ".*cycle_templates.*"}); } if (continue)

Answer: That code is not pretty. Why not simply catch the Exception? That is how they are meant to be used. So something like:
try {
   findANDclick(new String[]{"id", "menuButton"});
   findANDclick(new String[]{"src", ".*homeicon_calendar.*"});
   findANDclick(new String[]{"src", ".*cycle_templates.*"});
}
catch (Exception e) {
}

If any of the findANDclick calls fail, the subsequent ones will not be executed...
Btw, continue is a reserved word in Java, so you cannot use it as a variable name.
Comment: Empty catch block?? You really mean to ignore NullPointerException, IllegalArgumentException, etc., with NO feedback to the developer?
Comment: The code does exactly what the poster asked for. He asked for a specific way to rewrite the control flow of code he already (almost) had and this is that way.
Comment: I updated my question to make clearer that the exception is handled and thrown inside findANDclick. So it will never ends with an error.
Comment: @Radek In Java when the method `throws` an exception the next method can't be executed. So, Mathias's answer deserves your attention, although it doesn't look like you expected.
Comment: @MockerTim: maybe I am not describing the screnario well enough. The findANDclick throws an exception inside `try`. Then it finishes and next findANDclick is executed. My question here is how to make sure in a `nicely readable` way that the 'parent' script that calls findANDclick will finish
Comment: @Radek See my updated answer/comment. I think that this solution is *clear* and is *shortest* one.
Comment: @Mathias Empty `catch (Exception ex)` blocks are the devil's work when debugging. Even in test code.
Comment: You point is valid but completely irrelevant to the question that was asked.
Answer: No, you cannot do this.
However, if the method throws an exception, why do you still need to check the continue flag then?  It will stop anyways if the findANDclick fails.
Comment: So the exception is handled inside the `findANDclick` method already?
Comment: Add it to your method declaration, something like this: `private void findANDclick(String [] args) throws Exception {`.
Comment: No, it will not stop. The `throw Exception` sends an error message to the log. The instance of findANDclick finishes but I want the 'parent' that calls findANDclick to finish as well. Is that clearer?
Comment: @wjans No! `findANDclick() throws Exception`. It doesn't handle it inside.
Comment: @MockerTim As far as I understand from the question and comments, in his scenario no exception is thrown, but there is an exception handled somewhere inside the `findANDclick`?
Comment: @Radek Ok, so you throw the exception yourself inside `findANDclick` but it's not propagated to the code where you call this method?
Comment: @wjans: it looks like. I don't know that I supposed to work this way :-) The `throw` is logged so I thought everything was ok.
Comment: If you do throw the exception correctly, and you don't catch it inside your `findANDclick` implementation, then execution should stop correctly witthout invoking consequent calls, and no checking for continue flag would be required.
Comment: @wjans: I thought you solved it :-) but if I don't catch the exception inside findANDclik I get `handled exception type Exception` error so I cannot run the script.
Comment: You should declare `findANDclick` with `throws Exception` and put all your calls to it in one `try` block.
Comment: I have all calls to `findANDclick in one try block. How do I declare it with `throws Exception`?
Comment: @Radek My example answers your last question. Read it attentively.
Comment: I know I'm a bit late to the party, but RFT automatically catches and logs exceptions if they propagate to the highest level. That's the source of this confusion.
Answer: Radek! Try to execute the following example, just to clarify how the Java exception handling works.
public class ExceptionHandling {
    /*Method that throws exception*/
    static void methodOne() throws Exception {
        System.out.println("methodOne();");
        try {
            throw new Exception();
        } catch (Exception e) {
            System.out.println("Exception caught in methodOne(), worked up, and thrown again.");
            throw new Exception();
        }
    }

    static void methodTwo() {
        System.out.println("methodTwo();");
    }

    public static void main(String[] args) {
        try {
            methodOne();
            methodTwo();
        } catch (Exception ex) {
            System.out.println("Exception caught in main()!");
        }
    }
}

Output of this example:
methodOne();
Exception caught in methodOne(), worked up, and thrown again.
Exception caught in main()!

It shows that the second method is never executed, if the first one throws an exception.
P.S. This should be a comment. But it's clearer in rich formatting.
Solution suggested by Mathias Schwarz
try {
   findANDclick(new String[]{"id", "menuButton"});
   findANDclick(new String[]{"src", ".*homeicon_calendar.*"});
   findANDclick(new String[]{"src", ".*cycle_templates.*"});
} catch (Exception e) {
   // Workup exception somehow.
}

has advantages to which you aspire (from Java developer's point of view):

It's short;
It's clear for Java developer.

Disadvantage is also clear: ugly exception handling construction.
But in Java you can't avoid it. It's language rules.
Comment: Your empty catch block is not a great idea. At the very least, throw and catch a specific subclass of Exception, so that programming errors like NullPointerException aren't accidentally caught.
Comment: but I call the findANDclick from `try construction`
Comment: and I have ` catch (Exception ex) {
                 logStoreException(ex);`
Comment: @Radek To make our help more productive, please, add [SSCCE](http://sscce.org) code to your question.
Comment: @etienne Right you are! Thanx. I'll fix it.
Answer: You can't use if. You can use a do/while loop that more or less provides the syntax you are looking for.
do { findANDclick(new String[]{"id", "menuButton"}); } while (continue)
do { findANDclick(new String[]{"src", ".*homeicon_calendar.*"}); } while (continue)
do { findANDclick(new String[]{"src", ".*cycle_templates.*"}); } while (continue)

BTW, like someone mentioned above, continue is a bad choice for a variable name.
Comment: Even worse, it will keep executing the same `findANDclick` as long as `continue` is `true`.
Comment: It won't work. I need the condition to be executed before the code. In your code all the findANDclik's are going to be executed.
Comment: oops. Should have thought a bit more before posting.
Comment: I was thinking about that too. I can update `continue` inside `findANDclick` so it would be ok and don't look for ever.
Answer: I think you should try this:
    String[][] params = {
        {"id", "menuButton"},
        {"src", ".*homeicon_calendar.*"},
        {"src", ".*cycle_templates.*"}
    };

    for (String[] param : params) {
        if ( findANDclick(param) ) break;
    }


How to read array posted from form repeater

Question: How to read array posted from form repeater
the best is the array got from a form post..
Array (
    [products] => Array (
        [0] => Array (
            [pid] => 1
            [qty] => 2
        )
        [1] => Array (
            [pid] => 1
            [qty] => 2
        )
        [2] => Array (
            [pid] => 1
            [qty] => 2
        )
    )
)

my question is how to read value like this:
$pid[1]= 1
$qty[1] = 1
Comment: `$thePid = $arr['products'][0]['pid'];`
Comment: you need to use $products = $_POST['products']; echo $products[0]['pid'];
Comment: @RiggsFolly   -- thanks for guiding  but nothing works.
Comment: @Girish Sasidharan - thanks for guiding but did does not works.
Comment: Nothing working, and yet you accept an answer that say exactly what I did
Comment: Sorry MR.RiggsFolly  @RiggsFolly the answer that i accept is more clear than yours...   iam not a developer so i cant understand the way you answerd.
Answer: You can access the array values like this.
<?php
    $array = array("products"=> [["pid" => 1 , "qty" => 2 ],
                                 ["pid" => 2 , "qty" => 3 ],
                                 ["pid" => 3 , "qty" => 4 ]]);
    echo "PId: " . $array["products"][0]["pid"] . "
";
    echo "qty: " . $array["products"][0]["qty"] . "

";
    print_r($array);
    ?>

see the PHP fiddle for the working version
Comment: ```  echo $_POST["products"][0]["pid"]; ```