Por que a resposta de request HTTP não reconhece caracteres especiais?

Question: Por que quando eu faço um request usando BeautifulSoup em Python, a minha resposta não considera caracteres latinos?
Código:
import requests
from bs4 import BeautifulSoup
req = requests.post(url= f"https://www.linkcorreios.com.br/?id=LE132696585SE")
soup = BeautifulSoup(req.text,'html.parser')
texto = soup.find('ul', {'class': 'linha_status m-0'}).text
print(req)
print(texto)

Resposta:
<Response [200]>
Status: Objeto em trânsito - por favor aguarde
Data  : 29/04/2021 | Hora: 10:29
Origem: Unidade de Logística Integrada - Curitiba / PR
Destino: Unidade de Tratamento - Cajamar / SP

Estou usando o VSCode com Python 3.9.4
Answer: Resumindo, é um problema de encoding (se quer entender a fundo o que é um encoding, leia aqui).
Mas basicamente, toda a informação trafegada vai e vem na forma de bytes, que são transformados em texto e vice-versa. E existem várias formas de converter bytes de/para texto (vários encodings diferentes), e esse tipo de problema acontece ao tentar usar um encoding quando na verdade foi usado outro (por exemplo, se a resposta foi convertida para bytes usando um encoding, mas você tenta converter esses bytes para texto usando outro encoding).
Na documentação do módulo requests podemos ver aqui o seguinte:

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers.

E aqui:

When you receive a response, Requests makes a guess at the encoding to use for decoding the response when you access the Response.text attribute.

Ou seja, ao ser feita uma requisição, o requests tenta adivinhar o encoding da resposta, baseado nos headers HTTP. Então ao acessar o atributo text da resposta, será usado esse encoding para converter os bytes em texto. Mas o mesmo link acima ainda afirma:

The only time Requests will not do this is if no explicit charset is present in the HTTP headers and the Content-Type header contains text. In this situation, RFC 2616 specifies that the default charset must be ISO-8859-1.

Ou seja, se não houver um charset especificado nos headers HTTP, e o Content-Type contém text, o encoding será setado para ISO-8859-1. Podemos ver que esse é exatamente o caso, imprimindo os headers da resposta:
response = requests.post('https://www.linkcorreios.com.br/?id=LE132696585SE')
print(response.headers)

A saída foi:
{'Date': 'Wed, 23 Jun 2021 11:55:38 GMT', 'Server': 'Apache/2.4.7 (Ubuntu)', 'X-Powered-By': 'PHP/5.5.9-1ubuntu4.21', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Content-Length': '5090', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive', 'Content-Type': 'text/html'}

Podemos ver que o Content-Type é text/html (ou seja, contém text) e não há um charset explícito em nenhum dos headers. Por isso o encoding usado será ISO-8859-1, o que pode ser confirmado assim:
response = requests.post('https://www.linkcorreios.com.br/?id=LE132696585SE')
print(response.encoding) # ISO-8859-1

Portanto, ao tentar acessar o atributo text, os bytes foram convertidos para texto usando-se o encoding ISO-8859-1.
Mas como vimos, a página não está em ISO-8859-1, senão os acentos seriam mostrados corretamente.

A solução é simples: em vez de passar o text da resposta, passe o content:
import requests
from bs4 import BeautifulSoup

response = requests.post('https://www.linkcorreios.com.br/?id=LE132696585SE')

# *** Em vez de response.text, use response.content ***
soup = BeautifulSoup(response.content, 'html.parser')
#                             ^^^^^^^

texto = soup.find('ul', {'class': 'linha_status m-0'}).get_text()
print(response)
print(texto)

Assim, o Beautiful Soup lerá os "bytes brutos" (em vez de um texto que foi decodificado usando o encoding errado), e ele saberá lidar com isso, convertendo esses bytes em texto (mais detalhes aqui). Com isso, os acentos serão mostrados corretamente.


watershed segmentation implementation using scikit image

Question: I have image with certain overlapping. I want to segment it with watershed algorithm using distance transform but are not able to achieve it.
I am following the steps

distance transform on binary image,
then dilation and opening and finally
watershed.

I an new to this field, kindly help.
Answer: I'd start with OpenCV and something like this, but it's in C++ which isn't a very friendly language.  Alternatively, if you use python, here's a link that demonstrates watershed. PyImageSearch is pretty new-programmer-friendly and has some great tutorials on image processing with Python.
Comment: but point i got stuck is selection of markers, using distance transform,i found local maxima as marker and directly using them for  watershed, still not satisfacoty result with this operation
Comment: It's difficult to diagnose without seeing source code.


Predictive modelling strategy for spatial interpolation: complex data structure, small sample size, but n > p

Question: I apologise in advance for the long post, my questions are deeply interconnected so it felt wrong posting them as separate threads. Please note also that I edited this post to account for Florian Hartig's comments and questions.
Context and goal
I would like to build a model to predict the physical condition of a bird species in various areas of a city. EDIT: My goal is spatial interpolation, I want to estimate the species' physical condition for all woody patches of the city based on the values observed across the woody sites I sampled. I should also mention that these predictions will only be used to parametrise another model (to be weights for the nodes of a graph-based connectivity model).
My dependent variable (DV) is continuous (approximately normal). I have a fairly small sample size ($N$ ~ 420) but the good news is that I have few predictors ($p$ ~ 15) – which describe the environmental conditions surrounding the nestboxes where each bird was measured – and they're all biologically relevant.
Now, the trouble is my data has a complex structure and my observations are not entirely independent:

An observation is a bird's measurement in a given nestbox for a given year (i.e. $N$).
Observations were made over 4 consecutive years (2019-2022) and part of them were made on the same nestboxes. Therefore, while most of my observations were made on independent nestboxes (i.e. with a single measurement; $n_{single}$ ~ 240), the rest of my nestboxes are associated with 2 to 4 different bird measurements ($n_{multi}$ ~ 180).
Some of my nestboxes are spatially clustered but I'm not sure it's a problem because my predictors should account for this spatial autocorrelation while many observations are not clustered.

EDIT to answer Florian Hartig's first question (a): I think that I want to predict outside of my spatial structure since I want to interpolate values (i.e. predict between my sampled sites/nestboxes). For the temporal structure however, I don't really know since my data are static and have no temporal resolution (for a given nestbox, all predictors values remain the same every year), so I think I want to predict for an "average" year.

My edited questions
I'm quite new to predictive modelling so I'm trying to figure out what would be the best approach for me here in terms of model type and validation procedure, hence my questions:

Given my sample size and data structure, what method should I use to evaluate the predictive performance of my future model(s): held-out test set or resampling (cross-validation or bootstrap)? EDIT (answer): I asked a more general question here and I read Frank Harrell say somewhere that hold-out testing is unreliable for $N$< 20 000, so I guess I should use either repeated-CV or bootstrap to evaluate my modelling approach.
Whatever the method, how can I account for the particular structure in my data? Should I use a kind of stratified resampling? If so, any advice on how to do that using R? EDIT (answer): It seems I should use blocked-CV (or blocked-bootstrap?) to account for my data structure. The blockCV package could here be useful.
Alternatively, would using a mixed modelling framework enable avoiding this daunting stratification process? I could also drop the repeated measures but I'd prefer not losing observations. EDIT (answer): theoretically yes but that may not work well.
If all my observations were independent, I would not be in a high dimensionality problem but given my data structure, I don't know how to estimate my effective sample size. Should I go for variable selection (e.g. LASSO, AIC-based model selection and averaging) to obtain a better $n/p$ ratio?
Actually, considering my situation, can I afford any model optimisation (e.g. variable selection or hyperparameter tuning) and associated double-bootstrap/nested-CV? This answer (or this one) makes me think that I can't yet I'm unsure as I don't know the required sample size for these procedures as it seems there's no theoretical sample size limitations to CV folds (it only affects the precision of CV estimates). EDIT: Still, if I do model optimisation, I should still use blocking to account for my structure (e.g. giving something like nested blocked-CV?). Yet that seems awfully complicated with regards to my goal.
All in all, what model type and validation procedure would you choose in my place? And with a focus on which metric (R², MSE, AIC, etc.)? Initially, I thought about a mixed effects Random Forest regression to maximise prediction accuracy compared to a "regular" LMM, but simpler solutions are perhaps wiser?

As you can see, there are crucial gaps in my understanding so I'm having trouble connecting the dots and deciding on the safer path to wander. I really want to learn so I purchased "Regression Modeling Strategies" by Frank Harrell as well as "Statistical Learning with Sparsity" by Hastie et al. but it will take me months to read and digest them both. Consequently, any helpful advice and resources (especially for R implementations) will be highly appreciated.
Answer: OK, I will not be able to answer all your question but I will provide some thoughts
a) First of all, for structured data, you have to specify if you want to predict inside your structure, or outside. So, for example, if you have a random effect year, do you want to predict for a year in your dataset, or to a new year? Same with spatial structure and so on.
b) For all ML approaches such as bootstrap or CV, permutations and blocking should be done according to the answer to a), see also Roberts, David R., et al. "Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure." Ecography 40.8 (2017): 913-929. So, if you want to see how your model performs for predicting to a year for which you have data, use a random CV, but if you want to see how it would perform in predicting to a new year use a blocked CV.
c) If you could "guess" correctly the structure of your data, and use appropriate models (such as mixed models, spatial CAR structure etc.) you could in principle rely on the parametric inference of these models also outside your structure, without nonparametric methods such as a blocked CV, but some of our simulations in the Roberts paper suggest that this does not work reliably on real data, presumably because it is rare that you will guess the correct model structure.
d) I don't see a use for calculating effective sample size. It's poorly defined in such a context anyway.
e) Regarding model optimisation and validation: see answer a)
Response to edited question:
Hi Fanfoué, thanks for your clarifications - if you want to INTERPOLATE in space, you want to predict INSIDE your spatial structure, so no need for blocking in space, and if you want to predict for an average year, you can probably ignore time as well. If you are really only interested in predictions, just use a random forest or BRT with all environmental features + x,y as predictors, and a random CV to validate.
Comment: Thank you so much for your answers, it is much clearer now!


imageresizing.net plugin to allow image resizing/scaling/cropping

Question: Is there a simple (jquery) imageresizing.net plugin that would allow me to easily integrate image resizing/scaling/cropping in my asp.net webforms application?
Answer: Cropping, yes; jCrop does this easily. There is also a crop preview plugin we created for jCrop.
StudioJS is a more complex plugin that tries to do everything (filtering, rotation, etc), but is slightly less refined overall.


Flutter Image.network is not showing when using loadingBuilder?

Question: I'm using Image.network to load image from network. When I simply pass the url the image works fine. It will be displayed after few seconds. But when I use loadingBuilder and errorBuilder the images are loaded but not displayed. The circular progress I created inside the loadingBuilder completes and disappears but image is not shown.
Here code
Image.network(
      widget._imgLink,
      loadingBuilder: (context, child, loadingProgress) {
        return Center(
          child: CircularProgressIndicator(
            value: (loadingProgress != null)
                ? (loadingProgress.cumulativeBytesLoaded /
                    loadingProgress.expectedTotalBytes)
                : 0,
          ),
        );
      },
      errorBuilder: (context, error, stackTrace) {
        return Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Text("Error loading"),
            FlatButton(onPressed: () {}, child: Text("Retry")),
          ],
        );
      },
    ),

How can I tell the widget to display the image after its loaded?
Thank you.
Comment: you are not using `child` parameter at all in your `loadingBuilder: (context, child, loadingProgress) {`
Comment: Do I have to provide the child and how to use it?
Comment: see https://api.flutter.dev/flutter/widgets/Image/loadingBuilder.html
Answer: Returning child from loadingBuilder when loadingProgress is null will work.
Sample:
loadingBuilder: (context, child, loadingProgress) {
        if(loadingProgress == null)
          return child;

        return Center(
          child: CircularProgressIndicator(
            value: (loadingProgress != null)
                ? (loadingProgress.cumulativeBytesLoaded /
                    loadingProgress.expectedTotalBytes)
                : 0,
          ),
        );
      },

Answer: Use CachedNetworkImage which loads the image and provide error and loading widgets.
dependencies:
  cached_network_image: ^2.3.3

Code:
CachedNetworkImage(
  imageUrl: "http://via.placeholder.com/200x150",
  imageBuilder: (context, imageProvider) => Container(
    decoration: BoxDecoration(
      image: DecorationImage(
          image: imageProvider,
          fit: BoxFit.cover,
          colorFilter:
              ColorFilter.mode(Colors.red, BlendMode.colorBurn)),
    ),
  ),
  placeholder: (context, url) => CircularProgressIndicator(),
  errorWidget: (context, url, error) => Icon(Icons.error),
),

Comment: Thanks but why can't I use Image.network for the same as it provides these functionalities too.
Comment: CachedNetworkImage doesn't work in flutter-web so Image.Network still holds value.
Comment: you can use it, CachedNetworkImage is a more advanced way to way to show network image, it too provides caching.


Unable to see URL path setting menu for a user edit page

Question: I want to see "URL path setting" tab which appears while editing any content, but can't see it while editing a user. need help or suggestion regarding this.
thanks
Answer: Download & install Pathauto module & then check.


Inserting submenu to main menu using jquery - good idea?

Question: I have main menu showing only first level links (3 items).
<code>**Home**
**Themes**
-green
-blue
-red
**Jokes**
-long
-short
-medium
</code>
I am using Menu block module (to show second level items) + Power menu (to make menu items active if current node has certain term).
So when a node with taxonomy "theme" is displayed, the items under "themes" shows up in block menu.
What I would like to do is to somehow inject the block menu items inside primary menu. The main reason is to be able to use responsive menu, that immediately shows user menu options based on node term. (It is more of a usability idea for mobile users - not having to click through full menu).
Question: Do you think it is a good idea to inject such menu using jquery? Or is there a better drupal way of doing this? Thank you
Comment: Is it block menu module or Menu block module or Block per menu module, There is nothing called block menu module.
Comment: *The main reason is to be able to use responsive menu, that immediately shows user menu options based on node term.* So, do you want that the menu items gets changed based on the title of the node. Or do you want Main menu, where user just hover on the item or click on the item it opens one small block under that which shown the all options under that particular menu item, so user can see what that main menu items provide under it and user can navigate easily.
Comment: @CodeNext HI, I would like the menu to be changed/altered based on node's taxonomy - this change is happening already - due to the menu_block functionality. The last thing is to move the menu_block items, inside the menu.
Answer:
What I would like to do is to somehow inject the block menu items
inside primary menu.
Or is there a better drupal way of doing this?

TB Mega Menu allows you to put blocks and view blocks inside your menus. Perhaps you should consider using this module instead.


Is putting content in the codebase or the database better for performance?

Question: I recently created a website footer using a block template. Some of the footer's content was entered through the block's wysiwyg (stored in the database), but other code-heavy elements were placed in the block template file (stored in code). I've noticed that the footer block loads slow at times. Other times, the content inserted through the wysiwyg loads a few seconds before the content that was inserted in the the block template.
I'm curious, is storing html content in the database better for performance than storing html content in template files?
EDIT: After I posted this, I also realized that the footer is being rendered with Blocks whereas the rest of the page is being render with Panels. I'm not sure if that would have any effects of its performance.
Answer: Quick answer is that the DB should be slower. In reality the only way to know for sure is to profile the PHP code and get a cache grind of it. One being rendered by panels and the other by core blocks is the main reason most likely in this case.


Como configurar o GitHub no Visual Studio 2013 (Compartilhar projeto na rede Local)

Question: Como configurar o GitHub (Version 2.26) no Visual Studio 2013 (no VST 2013 em Solution Explorer/Team Explorer) para compartilhar um projeto na Rede local?
Comment: Veja se isso ajuda: http://michaelcrump.net/setting-up-github-to-work-with-visual-studio-2013-step-by-step/
Answer: O GitHub é um serviço de hospedagem, você não irá configurá-lo no Visual Studio! O que você tem que fazer é clonar o repositório e depois fazer o push para o GitHub.

Clique em Connect to Team Projects

Clique em Clone

Siga o processo normal do Git... Se você não tem experiência com Git eu escrevi um post Git básico…para quem está vindo do TFVC, ou Source Safe, ou SVN, tem 3 partes.
Comment: Muito obrigado Brandão! Vou Estudar teu post, realmente não tenho experiencia com o Git!


Como rodar um modelo para cada subset de dados de uma serie temporal por ver

Question: Eu tenho uma base de dados que tem informação de consumo de ração de animais (190 animais) a cada dia (88 dias).
Segue abaixo
table1 <- read.csv("Data.csv", header = TRUE)
table1

Animal Dia Consumo
1 1 245
1 2 256
1 3 300
1 4 450
2 1 245
2 2 256
2 3 300
2 4 450

x = table1$Dia

y = table1$Consumo

Animal=table1$Animal

Primeiro eu preciso rodar um modelo DLM (já tenho o programa) somente para um animal.
runDLM = function(beta) { x, y,........ }

Depois eu preciso rodar para todos os animais. Para todos os animais eu pensei no comando descrito abaixo, mas não sei se esta certo
for (Animal in 1:190) {
    runDLM(beta)
    }

Comment: Parece estar errado, a função `runDLM` não depende de `Animal`, não é um argumento da função, o único argumento é `beta`.
Answer: Inventei dados e uma função, mas pode usar a mesma lógica:
set.seed(123)
table1 = data.frame(Animal=rep(1:6,each=10),Dia=rep(1:4,15), Consumo=round(runif(60,240,500),0))

runDLM = function(beta,dat) {
  x=dat$Dia
  y=dat$Consumo
  coef(lm(y~I(x^beta)))
  }

sdat=split(table1,table1$Animal)
betat=2
sapply(sdat,runDLM,beta=betat)

#                         1          2          3          4          5          6
#    (Intercept) 355.804959 319.156589 376.897521 398.223256 313.494215 369.547287
#    I(x^beta)     5.322314   6.699225   3.538843  -2.167442   2.523967  -1.993798

Answer: Bom, como geralmente se fala em tutoriais de R, "se tá usando um for-loop, ta fazendo errado". Além deixar o código um pouco mais lento, vc não está utilizando um dos fatores mais importantes da linguagem, que é a sua orientação funcional, extremamente útil em problemas como o seu: aplicar funções complexas estruturadamente em dados.
A resposta do Robert está corretíssima. Ofereci apenas uma versão mais moderna utilizando os pacotes dplyr + tidyr, broom e purrr para:

gerar dados com a mesma estrutura que nos mostrou
estimar um modelo linear simples nos dados gerados
extrair os coeficientes estimados de cada modelo

A estrutura do meu código deve lhe ajudar à estimar os seus modelos sem problemas.
library(tidyverse)

## gerando dados com a mesma estrutura apresentada

set.seed(134)

df <-
  tibble(Animal = 1:190, taxa = runif(n = 190, min = 3, max = 23)) %>%
  crossing(dia = 0:3) %>%
  mutate(consumo = 245 + dia*taxa)

df

# A tibble: 760 x 4
   Animal    taxa   dia  consumo
   <int>     <dbl> <int>    <dbl>
1      1  7.007272     0 245.0000
2      1  7.007272     1 252.0073
3      1  7.007272     2 259.0145
4      1  7.007272     3 266.0218
5      2 15.551850     0 245.0000
6      2 15.551850     1 260.5518
7      2 15.551850     2 276.1037
8      2 15.551850     3 291.6555
9      3 20.629976     0 245.0000
10     3 20.629976     1 265.6300
# ... with 750 more rows

Agora vamos estimar um modelo linear simples com lm para a função consumo ~ dia e extrair os coeficientes de cada modelo com o auxílio da função broom::tidy, que extrai os coeficientes e estatísticas em um data.frame "tidy"(limpo):
## gerando um modelo linear no dia para cada animal e extraindo os coeficientes  --------------

df %>%
  group_by(Animal) %>%
  nest(.key = dados_animal) %>%
  mutate(model = map(dados_animal, ~lm(consumo ~ dia, data = .))) %>%
  mutate(coef = map(model, broom::tidy)) %>%
  unnest(coef)

Resultando em:
# A tibble: 380 x 6
   Animal        term   estimate    std.error    statistic      p.value
   <int>       <chr>      <dbl>        <dbl>        <dbl>        <dbl>
1      1 (Intercept) 245.000000 1.174950e-15 2.085196e+17 2.299886e-35
2      1         dia   7.007272 6.280370e-16 1.115742e+16 8.032903e-33
3      2 (Intercept) 245.000000 0.000000e+00          Inf 0.000000e+00
4      2         dia  15.551850 0.000000e+00          Inf 0.000000e+00
5      3 (Intercept) 245.000000 4.699798e-15 5.212990e+16 3.679818e-34
6      3         dia  20.629976 2.512148e-15 8.212086e+15 1.482836e-32
7      4 (Intercept) 245.000000 0.000000e+00          Inf 0.000000e+00
8      4         dia   8.025121 0.000000e+00          Inf 0.000000e+00
9      5 (Intercept) 245.000000 0.000000e+00          Inf 0.000000e+00
10     5         dia  15.198999 0.000000e+00          Inf 0.000000e+00
# ... with 370 more rows

Para aprender mais sobre o jeito "tidyverse" de se programar, recomendo dar uma olhada nos artigos no site do pacute e no livro R for Data Sceince, principalmente na parte sobre como trabalhar com muitos modelos, pois é basicamente como apresentei essa resposta.


Show that $f_ng_n \to fg$ in $B(X)$.

Question:
If $f_n \to f$ and $g_n \to g$ in $B(X)$, show that $f_ng_n \to fg$ in $B(X)$.

Note that given a set $X$, $B(X)$ is the vector space of all bounded, real-valued functions $f: X \to \mathbb{R}$ and we supply $B(X)$ with the sup-norm $||f||_{\infty} = \sup\limits_{x \in X}|f(x)|$.

Given that  $f_n \to f$ and $g_n \to g$ in $B(X)$; which is precisely $||f_n-f||_{\infty} \to 0$ and $||g_n-g||_{\infty} \to 0$. Thus we can find some $N \in \mathbb{N}$ such that for every $\epsilon > 0$, $$||f_n-f||_{\infty} < \frac{\epsilon}{2||g_n||_{\infty}} \ \ \ and \ \ \ ||g_n-g||_{\infty} < \frac{\epsilon}{2||f||_{\infty}}, \ \ \ \forall n\geq N. $$ Then we may compute the following: $$||f_{n}g_{n}-fg||_{\infty} \leq ||f_{n}g_{n}-fg_{n}||_{\infty} + ||fg_{n}-fg||_{\infty} \leq ||f_{n}-f||_{\infty}||g_n||_{\infty}+||g_{n}-g||_{\infty}||f||_{\infty}.$$ Therefore, $||f_{n}g_{n}-fg||_{\infty} < \epsilon$ so  $||f_{n}g_{n}-fg||_{\infty} \to 0$. It follows that $f_ng_n \to fg$ in $B(X)$.
Is this a correct argument for this problem? There was a lot of attention to detail when setting up and performing the triangle inequality, so I just want to be sure.
Comment: In the equation after "Then we may compute the following" the first term in the final bound should be $||f_n-f||_\infty ||g_n||_\infty$. You might also want to make clear that since $g_n \rightarrow g$ in norm, the norms $||g_n||$ are uniformly bounded and this is why an estimate of the form $||f_n-f||_\infty<\frac{\epsilon}{2||g_n||}$ for all sufficiently large $n$, is possible.
Comment: Notice that $g_n$ may be identically zero and thus $\|g_n\|$ can be zero, so dividing by it will not be possible.
Comment: @JonathanHole Awesome! Thank you for your feedback and time.
Comment: @jonathanHole - I figured this would be the problem, what if I defined $G = \sup_n||g_n||_{\infty}$ such that $||f_n - f||_{\infty} < \frac{\epsilon}{2G}$? Would that fix the error?
Comment: Yes, certainly!
Answer: First, since $f_n\to f$, in the $\infty-$ norm, then for $\varepsilon=1$, there exists an $N$, such that for $n>N$,
$$
\|f_n-f\|<1\quad\Longrightarrow\quad \|f_n\|< \|f\|+1.
$$
Hence
$$
\|f_n\| \le M=\max\{\|f\|+1, \|f_1\|,\ldots,\|f_N\|\}.
$$
Therefore
$$
\|f_ng_n-fg\|\le \|f_ng_n-f_ng\|+\|f_ng-fg\|\le \|g\|\|f_n-f\|+\|f_n\|\|g_n-g\|\\ \le
\|g\|\|f_n-f\|+M\|g_n-g\|\to 0.
$$

Show that $f_ng_n \to fg$ in $B(X)$.


When to use graph clustering (by constructing a graph from raw data) vs conventional clustering methods?

Question: This is a conceptual question. Say I have some tabular data, and a known similarity function i want to use to compare records in this tabular data. Records correspond to members of a MileageProgram, for example, and columns have categorical features corresponding to each member (Name, Membership tier, Country of Origin, City of residence, Color of hair, etc...). I could approach this in two (and there may be more, but I'm interested in comparing these two for the moment):
Approach 1:
One hot encode categorical variables (or find another way to embed/encode them). Use the known distance measure/function to calculate pairwise distances among the N members in my dataset. Then perform clustering using whatever makes sense for the structure of the data (e.g. K-means, DBSCAN, whatever...). Maybe throw in some dimensionality reduction
Approach 2:
Use the known distance measure/function to calculate pairwise distances among the N members in my dataset. Apply a threshold to create linkages based on these calculated distance values, and create linkages when distances are lower than some threshold T. Employ community detection methods on graphs (correct one, TBD).
Is there a rule of thumb to understand when to prefer Approach 1, and when to prefer Approach 2? What are the pros and cons of choosing one approach vs the other? I can see why thresholding might be a coarse step (in Approach 2) but there must be some scenarios where Approach 2 is the better approach to take?
Comment: I dont actually have a dataset in hand, just trying to come up with a conceptual understanding of why one approach may be superior, and in what scenario. So ideally the approach would not depend on a dataset having only categorical variables, or if you'd pick Approach 1 over 2 when you only categorical variables and Approach 2 when you have a mixture, that would be interesting to understand why as well.
Comment: Where do you draw the line between "conventional" and "graph" clustering? Is hierarchical clustering (which constructs a tree, which is a type of graph) conventional? Why is DBSCAN "conventional"?
Comment: Do you have only categorical variables?
Answer: The simple answer is "it depends on your data and the question that you are posing about your data". Furthermore, "it may depend on who is your audience".
I understand that you split the literature in two groups that do not "talk" very  much with each other (different periodicals, books, conferences...).
First approach: the classical and conventional approaches used in Machine Learning/Statistics.
Many possible different solutions, but as far as I know, the most popular ones are based on similarity metrics such as Euclidean distance, Cosine...
Second approach: more common among people that belongs to the field of Networks/Complex Networks/Social Networks. Some possibilities are here.
Most solutions are based in two different concepts: (1) Modularity (2)
Surprise (you do not need actually to evaluate the distance here - are you thinking about something else?):
Let me try to defend my assertion "it depends on your data and the question that you are posing about your data" through examples:
1) Imagine that you have a network of airports in some country and the only data that you have is the flow from one airport to the other and the intention of your work is to identify "clusters" or "communities" among these possible collection of airports (for business proposals you want to know whether the average person that lives in the north of USA travels in her/his neighborhood or not) For me the most natural approach in this case would be the second approach. Note that there is no "natural similarity (distance)" to be associated with two airports in this network.  We can build (I actually did something very related to this - although the focus was not to find exactly community structure), but maybe you understand what I am trying to say: 10 people in the same room may build a different metric and some of them may not make any sense for you or for me. Anyway, it may not be natural to everyone, but we can build a similarity function and we can use the first approach.
2) Your example… Imagine now that you have a some data and a clear similarity function. I would try a method of the first approach. Because you have all you need at hand and it is more natural.
However, the things are not so simple. There are some cases that both approaches may be natural. In the end, you will need to "sell" the technique you used for someone (your boss, your advicer, yourself). It will be much easier if you use the technique your community is more used to applying in similar situations.
If you are like me and you do not like the answer that I am giving to you, I would try to build simulate data (monte carlo simulation) that try to cover all the aspects of your data and compare the methods. However, is it possible to "model" all the aspects of the data? What is the generator process of the data? This is another difficult question.
I have not finished yet.
You compared two big fields of clustering/community detection.
Minor differences in your data or the similarity distance that you used will return different clusters. Let me give another example:
3) As far as I know when people use the K-means usually consider the Euclidean Distance. However, if you go to the field of Natural Language Processing people prefer to use the cosine similarity because in this field they use the cos similarity among vectors of TF-IDF, that makes much more sense than the Euclidean distance in this case. Although there is a relation between the Euclidean Distance and cos similarity, you have to change the algorithm. In this case, you have to use the Spherical K-means.
Answer: I'm not sure whether I share your understanding of "approaches". In my understanding, DBSCAN should belong to "Approach 2", as it applies "a threshold to create linkages based on these calculated distance values, and create linkages when distances are lower than some threshold T". So I offer my answer according to my understanding and hope it helps.
If I were to "cluster" the clustering algorithms into two kinds, they would be:

"Connecting algorithms", versus
"Separating algorithms".

The names are chosen to emphasize the difference between the two, and not necessarily to describe their inner workings. Also, such a taxonomy is an oversimplification; in reality, most algorithms fall somewhere between the two extremes.
"Connecting algorithms" understand clusters as collections of observations being "similar" by some similarity measure, and thus "connecting" them into clusters. The simplest and most prototypical of these is K-means, where the similarity is expressed as "having a low distance from the cluster centroid". Agglomerating hierarchical clustering, which connects observations according to their pair-wise similarity, would be another example.
"Separating algorithms", in contrast, understand clusters as collections of observations being dissimilar (separate) from other clusters. Spectral clustering
 is probably the most prototypical for this group, as it explicitly cuts the graph along the connections with greatest separation power.
Now, when to use which type of clustering depends on your data, your similarity measure and the desired information you want to derive. As I have done some work on spectral clustering, I'm somewhat biased towards them and consider them superior in terms of achieving "good" clustering results. The drawback is that the results are hard to interpret: For each cluster, you get a list of points that belong to it, but you can't really see why it is so. On the other extreme, K-means offers a straightforward interpretation: Points in each cluster are most similar to its centroid (the "platonic ideal" of that cluster, if you want). The drawback is that it only works when your clusters are convex and symmetrical (both according to your similarity measure) around the centroids.
For your concrete problem, where you have categorical variables, I believe graph-based approaches are more appropriate than centroid-based. Computing the mean (or any kind of centroid) for categorical variables is a very unnatural thing to do, but you can naturally construct a graph by connecting the observations based on the shared values of their variables, and weighting the connections accordingly.
Comment: Traditionally k-means is regarded a "partitioning" clustering method, which seems to be semantically closer to your "separating algoritms". Indeed, k-means ties points to centroids, but centroids are functions of clusters; separated clusters must already be defined in order to go further apart during the re-assignments.


Modifying the date column calculation in pandas dataframe

Question: I have a dataframe that looks like this

I need to adjust the time_in_weeks column for the 34 number entry. When there is a duplicate uniqueid with a different rma_created_date that means there was some failure that occurred. The 34 needs to be changed to calculate the number of weeks between the new most recent rma_created_date (2020-10-15 in this case) and subtract the rma_processed_date of the above row 2020-06-28.
I hope that makes sense in terms of what I am trying to do.
So far I did this
def clean_df(df):
    '''
    This function will fix the time_in_weeks column to calculate the correct number of weeks
    when there is multiple failured for an item.
    '''

    # Sort by rma_created_date
    df = df.sort_values(by=['rma_created_date'])

Now I need to perform what I described above but I am a little confused on how to do this. Especially considering we could have multiple failures and not just 2.
I should get something like this returned as output

As you can see what happened to the 34 was it got changed to take the number of weeks between 2020-10-15 and 2020-06-26
Here is another example with more rows

Using the expression suggested
df['time_in_weeks']=np.where(df.uniqueid.duplicated(keep='first'),df.rma_processed_date.dt.isocalendar().week.sub(df.rma_processed_date.dt.isocalendar().week.shift(1)),df.time_in_weeks)

I get this
Final note: if there is a date of 1/1/1900 then don't perform any calculation.
Answer: Question not very clear. Happy to correct if I interpreted it wrongly.
Try use np.where(condition, choiceif condition, choice ifnotcondition)
#Coerce dates into datetime
df['rma_processed_date']=pd.to_datetime(df['rma_processed_date'])
df['rma_created_date']=pd.to_datetime(df['rma_created_date'])

#Solution

df['time_in_weeks']=np.where(df.uniqueid.duplicated(keep='first'),df.rma_created_date.sub(df.rma_processed_date),df.time_in_weeks)

Comment: I added an example of the output I should expect. Does that make it more clear?
Comment: Where does 15 come from?
Comment: As you can see what happened to the 34 was it got changed to take the number of weeks between 2020-10-15 and 2020-06-26.
Comment: The 15 came from seeing that oh hey looks like I had another failure at 2020-10-15, okay, hmm, I need to adjust the 34 then and take the number of weeks between that new date 2020-10-15 and when we finished the first failure rate so the date above that row in the rma_processed_date column. Does that sense?
Comment: Could you please try `df['time_in_weeks']=np.where(df.uniqueid.duplicated(keep='first'),df.rma_processed_date.dt.isocalendar().week.sub(df.rma_processed_date.dt.isocalendar().week.shift(1)),df.time_in_weeks)`
Comment: That seems to work. Will it work for more than 2 records though?
Comment: I actually tried it one something else and it does not seem to work, I will update the post with the example.


Java: Splitting Code into its atomic representation

Question: In Java, only few things are considered to be atomic (What operations in Java are considered atomic?). For example i++ consists of 3 different atomic operations: Load i into a register, add 1 to that register, write the new value of the register back to i. Something like that.
My question: Is it possible to "parse" Java Code into a sequence of it's atomic representation? So when input is "i++" i don't want to have i++ as output, i want to have "LOAD I TO REGISTER", "ADD 1 TO THAT REGISTER", "WRITE I BACK FROM REGISTER". Is that possible?
Googling didn't help much on that topic.
Comment: You could disassemble the byte code.
Comment: @ Thorbjørn Ravn Andersen: Now that i think of it. Isn't the byte code a 1to1 representation of the actual source code atomic written? Actually the bytecode should be what i need?
Comment: @ThorbjørnRavnAndersen one bytecode instruction may be composed of several assembly instructions.
Comment: Then you can ask hotspot to show the machine code instructions generated
Answer: In Java you don't have register access or assembly code access. The only way you can realize the above is to use Atomic classes as provide in the java.util.conrrent.atomic package. However, that would be more costly than a simple i++ unless you are doing this on a multi-threading background.
Answer: Yes and no. Short answer: it'll be very very hard.
The code will go through several steps before it arrives at the hardware where you could 'parse' the atomic representations.
It will first be compiled into bytecode. This is already closer to machine code, but not yet what you want. You can read it by opening the class file in an editor that can parse bytecode (Eclipse IDE can do this).
When you run the program, the Java Virtual machine will execute your bytecode. It may, depending on the situation, execute the bytecode as a scripting engine, compile it to machine code or change the bytecode and then execute it to get better performance. What the Java Virtual Machine does will depend on the system, the current load on your system and the bytecode. So it's actually impossible to predict the actual machine code it would generate before hand.
For a specific run of code, you can use an assembly debugger (disassembler) such as IDA. It will allow you to view what is being executed on the system as assembly code. You will most likely see a lot of code that is part of the Java Virtual Machine and not of your program. This will make things even more difficult.


which of the following vectors are in the span

Question: I have been asked to determine whether or not the following vectors are in the span of (1,0,2,1) , (-2,3,-1,1), (2,-2,1,-1)
a)  (1,1,2,1)
b) (1,0,0,0)
c) (0,1,0,0)
d) (0,0,0,0)
so here are my answers, vectors a and d are in the span, however, the vectors b and c are not in the span.
If we wrote down a system of linear equations that corresponds to the vectors, it would look like the one below. (i'll use vector b as an example)
x - 2y + 2z = 1
3y - 2z = 0
2x - y + z = 0
x + y - z = 0
i used gaussian elimination to find the solutions for the system of equations, and the solutions i found are x = -1/3, y = 4/3, z = 2...... however, when i plugged these solutions into the equations above, i got an inconsistency on the fourth equation (1 = 0), which tells me that the vector is not in the span,
did i make any mistakes somewhere? my professor did not give me the solutions to these problems so there's no way i can check if i did this correctly
Comment: It isn't clear where your coefficients are from in the system of linear equations. For example, why does the first equation have $3z$ instead of $2z$? Why does the second equation have $y$ instead of $3y$?
Comment: Sorry, i have corrected the equations.
Comment: That looks better, although I think the last equation should have $-2z$, not $z$. Also, have a look at this [wonderful guide to formatting mathematical expressions](http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference).
Comment: darn it, i made one last correction to the vectors. It actually is supposed to be -z and not -2z. sorry about that. so do you think i got the correct answers?
Comment: a. is the combination of $1(1,0,2,1) +1 (-2,3,-1,1)+1 (2,-2,1,-1)$\\

b. is not a combination of the given vectors;\\

c. is got with $\{0,\;1,\;1\}$;\\

d. is got with $\{0,\;0,\;0\}$.
Answer: A few observations:
Some of your work is right; some of it isn't. I notice that your system of equations had the wrong coefficients, even after you updated it, so I wonder if you are working too quickly or not paying close attention to the numbers? Unfortunately, with this kind of task, one wrong number will throw everything off.
Also, note that you shouldn't have to do any calculations at all for $(0,0,0,0)$, since this is trivially in the span. (It is $0\Bbb v_1 + 0\Bbb v_2 + 0\Bbb v_3$ where the $\Bbb v_i$ are your given vectors.)
Finally, it doesn't really make sense to say that you solved a system and got $(x,y,z) = (\ldots)$ but that at the same time you found that the system is inconsistent. If the system is inconsistent (which it is in this case), then there is no solution for $x,y,z$.
Comment: Yes you are right, i guess i was working too fast. The process of gaussian elimination should've made it obvious that there were inconsistencies. but my mistake was not finishing the elimination. Thank you very much


Two github accounts conflicting on the same computer

Question: I have 2 github accounts, one for personal use and one for work.
I am working on a personal website and so I have logged into git from the terminal using my personal username and password. When I do a 'git config --list' I am shown the correct username and password for my personal account as I would expect.
However when I then try and push my code up to my personal account it gives this error...
remote: Permission to <personal_username>/<personal_repo>.git denied to <work_username>.

I have no idea where it is even picking up my work username or work credentials from as they are not listed in my git config.
Please help!
Thanks
Comment: Are you using HTTPS or SSH for your repositories?
Answer: If the URL is an HTTPS one, check your git config credential.helper: if manager, if would have cached, you need to reject its URL
git credential-manager reject
protocol=https
host=github.com

Comment: Where do I put that code sorry? I've run a 'git config -l' and my work credentials aren't even showing there so I don't know where the computer is picking them up from anymore
Comment: @brianmcgowan credentials have nothing to do with git config. That command is to be typed in a bash or CMD, then you type `protocol=...` and `host=...`. What OS are you on? With which version of Git?
Comment: git version 2.17.2 (Apple Git-113) and I am using a Mac
Comment: @brianmcgowan OK. Could you upgrade to 2.27?


How to load Scripts async with Angular

Question: As I have some bad experiences with the mix of Angular and jQuery, I am trying to avoid jQuery in Angular. Right now, I should add some code to my Angular project but I am not sure how I should convert this. The code is coming from someone else who wants me to implement this in the project.
Within my project I have added all of the required libraries into the index.html. I think this is the only place where a library can be added.
Does someone know (if there is a way) how I can convert those jquery code lines into Angular or javascript code? Where in the project should I add this?
$('head').append( '<meta name="PAGENAME" content="$pagename">' );

$(document).ready(function() {

    $.getScript( "https://www.website.com/crsc/scripts/script.js" )
      .done(function( script, textStatus ) {
        console.log( 'loaded test' );
      })

      .fail(function( jqxhr, settings, exception ) {
               console.log('not loaded');
      });

});

Comment: Are you asking how to load a script in angular? If so you should update the question title.
Comment: you can add a script tag in your index file with async=true
Comment: I don't know why you are getting downvoted, it was a good question to me.
Comment: Possible duplicate of [How to use jQuery in AngularJS](https://stackoverflow.com/questions/22666289/how-to-use-jquery-in-angularjs)
Comment: @DanteTheSmith OP is **not** trying to use jQuery with Angular but rather the opposite, avoid using jQuery.
Comment: I think the question was not formed the right way. Because I'm a bit desperate for a solution, I didn't take a lot of time for the formulation. I'm sorry for that.
Answer: You can use  app.run which will run like a main method before the app loads everything
     app.run(["scriptService"],function(scriptService){
        scriptService.GetScripts()
          .then(function(success){console.log("loaded test");},
         function(error){ console.log("not loaded");}
    }]);

Next all you have to do is make a script Service.
    app.factory("scriptService", ["$http", function ($http) {
                this.GetScripts = function ()
    {
return $http.get(""https://www.website.com/crsc/scripts/script.js") .then(function(result) { return result});

    }

but in reality why do you not  just import the script through a script tag?
Loading a script async, since op didnt know that was an option I thought I would include it here ww3 schools async loading

definition and Usage The async attribute is a boolean attribute.
When present, it specifies that the script will be executed
  asynchronously as soon as it is available.
Note: The async attribute is only for external scripts (and should
  only be used if the src attribute is present).
Note: There are several ways an external script can be executed:
If async is present: The script is executed asynchronously with the
  rest of the page (the script will be executed while the page continues
  the parsing) If async is not present and defer is present: The script
  is executed when the page has finished parsing If neither async or
  defer is present: The script is fetched and executed immediately,
  before the browser continues parsing the page Browser Support The
  numbers in the table specify the first browser version that fully
  supports the attribute.
Comment: exactly a simple script with async load should be enough .
Comment: I thought so too, but i figured op had his/her reasons
Comment: I didn't know async loading would give me the same result, thanks

How to load Scripts async with Angular


Finding the asymptotes of a hyperbola

Question: Given the hyperbola $x^2/16$ - $y^2/9$   $ = 1$ what would the equation of the asymptotes be?
Comment: http://en.wikipedia.org/wiki/Hyperbola
Answer: Hint: Any hyperbola of the form
$$\frac{x^{2}}{a^{2}} - \frac{y^{2}}{b^{2}} = 1$$
has asymptotes given by the lines $$y = \pm \frac{b}{a} x$$
In your case, $a = \sqrt {16} = 4$, and $b = \sqrt{9} = 3$.
Comment: Votes have become so scarce! +1
Answer: The asymptotes are
$$y=\pm\frac34x$$
Now your work is to justify why....

Finding the asymptotes of a hyperbola


Is a Major Interval the same as a Pure Interval?

Question: I've been making electronic music for the past year but recently decided that I need to sit down and learn some actual theory. So, I picked up a copy of Guide to the Practical Study of Harmony by Tchaikovsky and I had a quick question about how he's describing major/pure intervals.
For example:

Thus, we learn, that a major or a pure interval is converted into a minor, diminished or augmented interval by lowering or raising the upper tone...

In this passage, and a few others in the introduction, he seems to be using Major and Pure interchangeably in his sentence.
So, I'm just looking for clarification here, are Major and Pure intervals the same thing? Or, I suppose this is possible, that the rule he's describing just applies to both intervals and that's why he's using the terms interchangeably in a sentence?
Answer: Nevermind. I found the answer here.

A perfect interval identifies the distance between the first note of a major scale and the unison, 4th, 5th or octave.   Only those intervals can be given the extra attached name as "perfect".
Comment: I usually think of _pure_ intervals as [_just intoned_ intervals](https://en.wikipedia.org/wiki/Just_intonation). I'm not sure what Tchaikovsky had in mind here, but perfect intervals and pure intervals can't be the same thing in his thinking, since there is no such thing as converting a perfect interval to a minor interval.
Comment: The problem is that your question is about the word "pure" as used by Tchaikovsky, and your answer is about the word "perfect". I think Tchaikovsky probably does mean "pure" to mean "major", mainly because unisons, fourths, fifths, and octaves are not called "minor" when they are lowered by a half step, but major intervals are.
Comment: @DavidBowling - for me, as well, a pure interval is a just interval.  But from the context it's clear that Tchaikovsky mean "perfect interval" in its normal modern sense of simply being a unison, a fourth, a fifth, or an octave.  Of course, there's nothing really "perfect" about a tempered fourth or fifth, but that's another topic.....
Comment: @DavidBowling -- well, it says "_a major_ or a pure interval is converted into a minor, diminished or augmented interval by lowering or raising the upper tone".  I think he's saying- sloppily- that the major becomes minor and the "pure" becomes diminished when the upper tone is lowered a semitone, not that pure intervals also become minor.  It's in any case ambiguous and shouldn't be in a textbook.
Comment: @ScottWallace -- I get it now; Tchaikovsky is giving major and perfect (pure) intervals as two fundamental types to be modified....


Imports dinâmicos em Javascript

Question: Preciso importar dinamicamente os caminhos para os assets de uma aplicação, por exemplo:
var cliente = clienteX
import ../../caminho/${cliente}/caminho/arquivo.less
Não será mudado em tempo de execução, ele apenas puxaria de um arquivo de configuração, qual cliente ele puxará os assets, tendo em vista que o Core do sistema é o mesmo.
Pelo que pesquisei o import é estático, então queria alguma ajuda ou ideia de algum workaround para essa situação.
Edit 1: Exemplo de estrutura

Sistema

Clientes

cliente 1

css

main.css

cliente 2

css

main.css

Exemplo de imports:
require ('../../Clientes/'+ cliente + '/css/main.css');
import ../../Clientes/${cliente}/css/main.css;
Comment: Já tentou usar o `require`? Se você está usando o `babel`, ele tem suporte para o `require` no browser. Exemplo: https://github.com/pedrolaxe/js-terminal/blob/master/src/js/terminal/commands/index.js#L20
Comment: @GabrielKatakura Já tentei também e não rolou muito bem :/
Comment: Podes tentar usar o `path.resolve` combinado com o `__dirname`, dá um exemplo da estrutura de diretórios para ficar mais claro
Comment: @Sergio atualizei ali na pergunta
Comment: Os exemplos que colocaste foi o que tentaste e não funcionou?
Comment: @Sergio isso ai
Answer: Você não consegue criar imports dinâmicos. Como você comentou, o import do javascript é estático, logo não aceita interpolação de strings.
O que você consegue fazer é usar o require.
require ('../../Clientes/'+ cliente + '/css/main.css');

ou
require (`../../Clientes/${cliente}/css/main.css`);

Devem funcionar. Caso não funcionem o seu erro deve ser outro. Consegue verificar se o path está correto? Se fizer o mesmo require estático com o caminho de um dos clientes ele funciona?
Exemplo:
require (`../../Clientes/cliente1/css/main.css`);

Nesse caso, consegue verificar se o valor da variável cliente está correto?
Caso não consiga importar estaticamente, verifique a configuração do seu compilador (Babel) para checar se não existe algum erro no loader.
Caso continue errado, consegue postar o log do erro no terminal?
Comment: Tentei com o require, mas igual não funcionou. Ele não retorna erro, simplesmente não importa. Fiz o require passando o caminho de um dos clientes e funciona. ai passei o nome do cliente pra uma variável e passei ela e ele não importou :/
Comment: Você consegue confirmar se ele realmente não está importando? Coloque um `body { background: #000 !important; }` no estilo importado só para validar que ele realmente não está sendo importado. Fez o teste importando sem a variável com o cliente1 no lugar? Caso continue dando erro, você consegue importar qualquer outro arquivo css no projeto? Para validar que não é um problema com o loader.
Comment: Sim, eu importando passando o nome do cliente direto ele importa e altera o css, se eu passar ele como variável ele não importa mesmo.
Comment: Consegue dar um `console.log` na variável? Para verificar que o valor dela é o esperado?
Comment: retorna a string do cliente mesmo hahaha. Parece que somente o fato de ter algo "variável" no caminho, faz com que ele não seja importado
Answer: Você consegue fazer imports dinâmicos sim, porém, até onde eu sei, apenas consegue fazer com arquivos '.js'. Vamos lá, suponha que você tem um arquivo muito grande e, dependendo do clique do usuário, deve fazer algo que é exportado por esse arquivo. Se você importar ele durante a requisição da página, pode ser que seu carregamento fique lento e não tenha o comportamento desejado, para isso existem imports dinâmicos. A idéia é que você importa o arquivo, e de dentro dele usa as funções de são exportadas.
Use assim:
const file = await import('./caminho')
const something = file.exportedFunction()

https://javascript.info/modules-dynamic-imports
Comment: Por favor, adicione mais detalhes para expandir sua resposta, como um código que funcione ou citações de documentação.

Imports dinâmicos em Javascript


Recursion and divisibility by $2^n$

Question:
A team plays a series of games, each of which results in either a win (W), a draw (D), or a loss (L). Let $S_n$ denote the number of possible sequences for a team which never loses two successive games in a series of $n$ games. Prove that $S_{2n}$ and $S_{2n + 1}$ are both divisible by $2^n$.
(Source: Singapore-Cambridge GCE A-Level N2007/9810/01/Q9)

I was able to come up with a recursive formula for $S_n$: Suppose the team lost the $n$th game. Then the team must have either won or gotten a draw in the previous $n-1$th game. There are $2S_{n - 2}$  possible cases.
Suppose instead the team did not lose the $n$th game. Then the team must have either won or gotten a draw for the $n$th game. There are $2S_{n-1}$ possible cases.
Hence, by the Addition Principle, $S_n = 2S_{n-1} + 2S_{n - 2}$.
However, I'm stuck at the second part. It could be possible to use induction (in fact, I'm sure this is the "correct" way), but I was looking at a combinatorial proof instead.
Comment: Yes, use induction on $n$ to show that $S_{2n}$ and $S_{2n+1}$ are divisible by $2^n$.
Comment: For the closed form, you'll have to study linear recurrence relations. https://en.wikipedia.org/wiki/Recurrence_relation#Linear_homogeneous_recurrence_relations_with_constant_coefficients
Comment: The closed form is $$S_n=\frac{(2+\sqrt{3})(1+\sqrt{3})^n - (2-\sqrt{3})(1-\sqrt{3})^n}{2\sqrt{3}}$$ Not how that helps to find a combinatorial answer. Basically, it is the coefficient of $\sqrt{3}$ when you expand $(2+\sqrt{3})(1+\sqrt{3})^n$.
Comment: @ThomasAndrews Thank you :) Could you explain the relationship with the expansion of $(2 + \sqrt 3)(1 + \sqrt 3)^n$? It's my first time coming across this and it's really interesting :)
Answer: Define an equivalence relation on the set of permitted sequences of $n$ win/lose/draw if the locations of the losses are the same.
If there are $k$ losses, then the equivalence class has $2^{n-k}$ elements, since the other times can be either win or draw, freely chosen. But $k\leq \left\lceil\frac{n}{2}\right\rceil$ so the number of elements in each equivalence class must be divisible by $2^{ \left\lfloor\frac{n}{2}\right\rfloor}$ since $\left\lceil\frac{n}{2}\right\rceil + \left\lfloor\frac{n}{2}\right\rfloor = n$.
So $S_n$ must be divisible by $2^{ \left\lfloor\frac{n}{2}\right\rfloor}$, which is what you are trying to prove.
You can go somewhat further.
When $n$ is odd, there is only one way to get the maximum number of losses, so the largest power of $2$ that divides $S_n$ is $2^{\left\lfloor\frac{n}{2}\right\rfloor}$.
When $n$ is even, there are $n/2+1$ different ways to get the maximum loses, and therefore when $n$ is divisible by $4$, $S_n$ is divisible by exactly $2^{n/2}$, while if $n$ is not divisible by $4$, $S_n$ is divisible by $2^{n/2+1}$, and possibly higher powers of $2$.

Recursion and divisibility by $2^n$


Do sth specific if an optional parameter (TRUE) is passed in the function call

Question: I'm coding a function for freecodecamp and I can't quite figure out how to pass a second optional argument (in this case it has to be True) and print something different depending if the optional arg is present or not.
instructions are : Create a function that receives a list of strings that are arithmetic problems and returns the problems arranged vertically and side-by-side. The function should optionally take a second argument. When the second argument is set to True, the answers should be displayed.
PS: I don't want to cheat, just can't figure out the 2nd arg part
I tried using
def arithmetic_arranger(problems, *args):
but I don't know how to use the conditional to indicate the function to show the result if 2nd arg is True
code so far is [https://github.com/Tarufetti/practica-public-/blob/f4a5e09c7f755d50b588575c6199188470408da9/arithmetic_formatter(NEWedit)]
Comment: Asterisk notation is when you want to allow _any_ number of parameters. It isn't for optional parameters. Also, your question must be self-contained (we should not have to visit an external link to see your code) Please take the [tour], read [what's on-topic here](/help/on-topic), [ask],
and the [question checklist](//meta.stackoverflow.com/q/260648/843953), and provide a [mre]. Welcome to Stack Overflow!
Comment: Does this answer your question? [optional argument Python](https://stackoverflow.com/questions/62826483/optional-argument-python)
Comment: @Alejandro No, it won't. That's what the default value is for.
Comment: So sorry to bother you. if the function call has only the list of problems (and nothing else), won't python show the 'missing one argument' error?
Comment: How have you called the function (please provide a self-contained [mre])
Comment: @PranavHosangadi I'm sorry. I'm new to SO and I am yet to undestand the way things are done here. Thanks for the help. I promise to take the tour.
Answer: I think you need something like that:
def arithmetic_arranger(problems, bool_var=False):
    if bool_var:
       #do something if an optional True argument is passed


Prove that for positive integer n we have $(2^n-1)^2 \mid 2^{n(2^n-1)} - 1$.

Question: How to solve this?
$$(2^n-1)^2 \mid 2^{n(2^n-1)} - 1$$
Comment: Where did you get that problem from?
Comment: My teacher gave me this problem
Comment: (2^n−1)^2∣((2^n-1)+1)^(2n−1)−1
Comment: Maybe for all positive x we have x^2|(x+1)^x-1
Comment: UUhhh, in this situation, what does | mean??
Comment: @simplest_mathematics It's a standard notation for "divides".
Answer: The expression $$\dfrac{(x+1)^x-1}{x^2}$$ is an integer for all positive integers $x$. We have $$(x+1)^x-1 = x^x+\binom{x}{1}x^{x-1}+\cdots+\binom{x}{x-1}x+1-1 \equiv 0 \pmod{x^2}.$$


Linking Django with MSSQL error: The database driver doesn't support modern datatime types

Question: I have been trying to link my mssql database to Django. I am using windows 10 on my machine. When I run the Django server there is no problem. However, when I want to use InspectDB I am getting this error:
    File "C:\ProgramData\Anaconda3\lib\site-packages\sql_server\pyodbc\base.py", line 365, in init_connection_state
            "The database driver doesn't support modern datatime types.")
django.core.exceptions.ImproperlyConfigured: The database driver doesn't support modern datatime types.

My Django settings looks like this:
DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'HOST': 'my_host',
        'PORT': '1433',
        'NAME': 'database_name',
        'OPTIONS': {
            'driver': 'SQL Server',
            'extra_params': "Persist Security Info=False;server=my_host"
        },
    },
}

I don't really understand this engine idea. However, I tried other engines non of them worked. Error pops up when I run the server. With this engine not error araise when I run the server but it gives me the above mentioned error when I want to run: python manage.py inspectdb
any insght would be greatly appreciated.
Comment: Maybe this helps? https://github.com/michiya/django-pyodbc-azure/issues/161. Which version of pyodbc are you using and which version of Django?
Comment: django-pyodbc-azure: version 2.1 and Django 2.1
Comment: and pyodbc itself?
Comment: pyodbc: 4.0.25. I try to switch to different engine and get an error about login failed for the user
Comment: which OS?  If linux, have you already installed the sql server odbc drivers?
Comment: It's windows 10 as I mentioned
Answer: In general, if you're using mssql on linux, you'll have to install the sql server odbc drivers.  You'll have to do the same on windows clients, but that's a little more straightforward.  Once you've got the native drivers installed, update the DRIVER option to be ODBC Driver 17 for SQL Server.
Comment: I did that and now I am getting : django.db.utils.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'EMEA\\kmoh'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0); [28000]
[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'EMEA\\kmoh'. (18456); [28000] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0)")
Comment: If I run simple query to get me something out of Django it works just fine. Same username. I am using windows 10 as mentioned above.
Comment: Set the `USER` and `PASSWORD` options on the settings.DATABASES dictionary.
Answer: If you are using django-pyodbc-azure 2.0, the below issue may help.
https://github.com/michiya/django-pyodbc-azure/issues/142
Comment: Not really helpful I guess. I have seen it already. Thanks for the hint anyway
Answer: A Long time ago but that can help others as nobody presented a possible solution for Windows which is the case here.
In my case, I was using ODBC DSN entries to connect to the MSSQL databases from Django like the below code and I had the ODBC System DNS entry configured for using the native and old 'SQL Server' driver wrongly, the same which is being used in the question code and what is causing the issue,  even though I had installed 'ODBC Driver 18 for SQL Server'.
  'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': '',
        'HOST': '',
        'USER': '',
        'PASSWORD': '',
        'OPTIONS': {
            'driver': 'ODBC Driver 18 for SQL Server',

        }
    },

Instead, I should configure the ODBC entry to use the new Driver

On this thread question, it's not clear if they have installed a different driver but certainly, that drive configuration is what is causing the issue.


Opening a .bsm file

Question: I am trying to read the contents of a pascal.bsm file. What I am basically interested in is the system calls included in the file. I dont know how to open and read the file.
The file I'm trying to read is here
It is a file generated by a solaris system and includes the audit data which are not in easily readable form.
What should I do?
Thanks.
Comment: Did you ever find anything on Windows to handle the native format?
Answer: Just use praudit on Solaris:
<code>$ praudit pascal.bsm
file,1999-03-01 13:31:59.731 +01:00,
header,36,2,system booted,na,1999-03-01 13:31:32.219 +01:00
text,booting kernel
header,126,2,open(2) - read,,1999-03-01 13:32:19.241 +01:00
path,/etc/security/audit_control
attribute,100664,root,other,8388608,62781,0
subject,root,root,other,root,other,257,257,0 0 172.16.112.50
return,success,4
trailer,126
header,150,2,ioctl(2),fe,1999-03-01 13:32:19.241 +01:00
path,/etc/security/audit_control
attribute,100664,root,other,8388608,62781,0
argument,2,0x5401,cmd
argument,3,0xefffed7c,arg
subject,root,root,other,root,other,257,257,0 0 172.16.112.50
return,failure: Inappropriate ioctl for device,-1
trailer,150
header,137,2,close(2),,1999-03-01 13:32:19.241 +01:00
argument,1,0x4,fd
path,/etc/security/audit_control
...
</code>
Comment: @Goaler444 You are welcome. Feel free to vote up my answer which, despite being undoubtedly correct, had not a single vote after almost two years ...
Comment: any idea on how to open them on ubuntu please?
Comment: The simplest way would be to install VirtualBox on your Ubuntu box, then a VM with a Solaris distribution and use its `praudit` command. Alternatively, you might try compiling from its source code, but that might be a much complex alternative.

Opening a .bsm file


Express A^(-1) matrix as linear combination of A^2, A and I3

Question: I am a bit confused. How can I express A^(-1) matrix as linear combination of A^2,  A and I3?
This is the matrix A: [0, 0, 7], [1, 0, 3], [0, 1, 8]
I found A^2 to be [0, 7, 56], [0, 3, 31], [1, 8, 67]
Thanks
Comment: The characteristic equation of the matrix might help.
Comment: Welcome to MSE. Your question is phrased as an isolated problem, without any further information or context. This does not match [many users' quality standards](http://goo.gl/mLWc8), so it may attract downvotes, or closed. To prevent that, please [edit] the question. [This](http://goo.gl/PlJyVQ) will help you recognise and resolve the issues. Concretely: please provide context, and include your work and thoughts on the problem. These changes can help in formulating more appropriate answers.
Answer: $$
       A-\lambda I=\left[\begin{array}{ccc}-\lambda & 0 & 7 \\ 1 & \lambda & 3 \\ 0 & 1 & 8-\lambda
               \end{array}\right]
$$
The characteristic polynomial is
\begin{align}
       \mbox{det}(A-\lambda I)&=-\lambda^2(8-\lambda)+7-(-3\lambda)\\
 &=\lambda^3-8\lambda^2+3\lambda+7
\end{align}
Therefore
$$
                A^3-8A^2+3A=-7I \\
                A(A^2-8A+3I)=-7I \\
                A^{-1}=-\frac{1}{7}(A^2-8A+3I).
$$
Comment: Hey, I hope I'm not breaking any guidelines asking you this, but is there a way to reach you personally?
Comment: @iamsoboredofmathhh : Is there something in particular you wanted to know?


react-admin exclude "record" from the redux-form

Question: I have an issue when I send SimpleForm (edit) request with react-admin.
The request includes more parameters than I have in the form's fields.
For example I have form:
<Edit {...props}>
    <SimpleForm>
        <TextInput source="title_new" />
        <TextInput source="age_new" />
    </SimpleForm>
</Edit>

It includes only 2 fields but when I click "save" the request includes more fields.
I understood that those fields are coming from the GET_ONE request which fill the data from the DB.
GET_ONE:
{
title: 'title',
title_new: 'title-new',
age: 'age',
age_new: 'age-new',
}

The update request UPDATE:
{
title: 'title',
title_new: 'title-new',
age: 'age',
age_new: 'age-new',
}

I expect that the UPDATE will include only the forms fields (title_new and age_new) without the title and age fields that come from "record".
Those fields make me a lot of trouble on the API side and I want to avoid/exclude them from all the forms, basically I want to send only the form inputs with the SimpleForm inputs only.
Few solutions I have in mind:
1. "Altering the Form Values before Submitting" here
2. Manipulate the request in the restProvider
Both solutions are not good for me because I have many forms like that and the restProvider code will look bad. Also I don't want to "alter" any form I build.
PLEASE ADVICE.
Comment: By the way, I asked similar question on AOR previous version: https://stackoverflow.com/questions/48600487/how-to-exclude-fields-from-simpleform
Answer: This is the way react-admin works. If you want the UPDATE dataProvider verb to post only the changes fields (and probably send a PATCH rather than a POST), you have to do it in the dataProvider.
I'm not sure that the provider will look bad after the change: all you have to do is to alter the UPDATE verb. By default, it looks like (for the simple rest provider):
            case UPDATE:
                url = `${apiUrl}/${resource}/${params.id}`;
                options.method = 'PUT';
                options.body = JSON.stringify(params.data);
                break;

You just need to update it like so:
            case UPDATE:
                url = `${apiUrl}/${resource}/${params.id}`;
                options.method = 'PATCH';
-               options.body = JSON.stringify(params.data);
+               options.body = JSON.stringify(diff(params.data, param.previousData));
                break;

Where diff can be written as:
const diff = (previous, current) => lodash.pickBy(current, (v, k) => previous[k] !== v);

Comment: This looks like the correct way to handle it for all-requests rather than to handle it like `addUploadCapabilities`
Comment: @François Zaninotto: You modify `params.data` so you should change the verb from `PUT` to `PATCH`.
Answer: To selectively send just altered fields, use a diff function like this:
// in diff.js
import { transform, isEqual, isObject } from 'lodash';

/**
 * Deep diff between two object, using lodash
 * @param  {Object} object Object compared
 * @param  {Object} base   Object to compare with
 * @return {Object}        Return a new object who represent the diff
 */
const diff = (object, base) => {
  return transform(object, (result, value, key) => {
    if (!isEqual(value, base[key])) {
      result[key] =
        isObject(value) && isObject(base[key]) ? diff(value, base[key]) : value;
    }
  });
};

export default diff;

then:
// in dataProvider.js
update: (resource, params) =>
    httpClient(`${apiUrl}/${resource}/${params.id}`, {
      method: 'PATCH',
      body: JSON.stringify(diff(params.data, params.previousData)),
    }).then(({ json }) => ({ data: json })),


Nagios log check plugin that doesn't use diff

Question: we are using check_log from the nagios-plugins to monitor logs, but the problem is that with big logfiles, even if we rotate logs hourly, the plugin causes significant spikes in memory and cpu usage.
That is because check_log is using diff to compare an old version of the log file with the current one.
Question is, without having to check the source of all check_log alternatives, is there any that doesn't use diff but for example follows the last checked linenumber and picks up checking from there?
Answer: check_logfiles plugin is what you're looking for:

When such a logfile rotation takes place, check_logfiles detects this
  and analyses the lines of the archived logfile, even if it's
  compressed.
Comment: Exaclty what I was looking for.
Answer: Checking log files on a scheduled basis is not a very efficient approach. While it would be posible to write a plugin which would would keep track of the position of the last offset in the file and handle rotations, a much simpler approach would be to run a filter daemon against the log files - there's lots of such tools available.
(you can still process the output of such tools using nagios for reporting / tracking purposes)
Comment: Imho running a daemon which prepares output for Nagios would add another layer of complexity (which also would need to be monitored).


Ignore paramter if field is null: Invalid token (

Question: I have a parameter in the where clause of my query I am trying to ignore if a field is blank.  Receiving an "invalid token (" error.
WHERE ....
AND
CASE WHEN trim (cltfst) || ' ' || (cltlst) is null THEN ''
    ELSE (('LWSI:first_status')='(All)' or TRIM (status) in ('LWSI:status_list'))
    END = 'TEST'

Says:  "Invalid token (, Valid tokens (" if that makes any sense!
Comment: That's probably your problem, it doesn't belong in the `ELSE` clause of the `CASE`. The ELSE clause of the CASE must return a value. It cannot be a logical expression.
Comment: What are you trying to do with `ELSE (('LWSI:first_status')='(All)' or TRIM (status) in ('LWSI:status_list'))`? What value are you trying to return there? It doesn't look like that `CASE` could ever return 'TEST'.
Comment: The "else" line is my parameter.  I am simply inserting the paramter selected by the user there.  It works when I use it in a "regular" where clause.  And you are right, I don't want "Test" returned at all.  I am just playing with syntax trying to get something to work.
Comment: So how to accomplish this?
Comment: Just use an OR in the WHERE clause of the select. Give us more information, and I can give you a better answer.
Comment: All I want to do is IGNORE the status parameter when cltfst/cltlst is null.
Comment: See my answer. I believe you are thinking about it incorrectly.
Answer: If you have two conditions for a query, lets call them condition_a and condition_b, and if condition_a is true, then you don't care what condition_b is (you ignore it). You can handle that in your WHERE clause with an OR like this:
select ...
from ...
where condition_a or condition_b

To ignore a logical condition if a field (field1) is blank, this would look like
select ...
from ...
where (field1 = '' or condition_a)
  and condition_x

In this case if field1 = blank, the condition is true no matter what condition_a is. condition_x will still have to be evaluated, but condition_a is effectively ignored if field1 is blank. But, if field1 is not blank, then condition_a must be true for the record to be selected. Once again, condition_x must be evaluated and also be true.

More specifically to your situation:
select ...
from ...
where ...
  and (trim (cltfst) || ' ' || (cltlst) is null or
        (('LWSI:first_status')='(All)' or TRIM (status) in ('LWSI:status_list')))

Not real sure what all your short hand means here, but it is ignored if trim (cltfst) || ' ' || (cltlst) is null.
Comment: Thank you Sir!  Sometimes a 2nd set of eyes is very helpful!


Google App Engine Node.js TLS 1.2

Question: Our application hosted on Google App Engine Node.js (Flexible Environment). We are now under review of security inspection and failing on the issue that Google App Engine supports TLS 1.0 and 1.1 versions.
Is there a way to enforce the use of only TLS 1.2? And also block ciphers that are below 128 bit?
Comment: Potentially of interest: https://stackoverflow.com/questions/42681247/can-google-app-engine-java-support-tls1-0?rq=1
Comment: That's my understanding as well. Maybe technically not exactly the load balancer, but somewhere around there, in the edge common infra serving all cloud clients.
Comment: I think I found more info [here](https://stackoverflow.com/questions/40853799/is-there-a-way-to-customize-nginx-proxy-in-google-flexible-appengine). It seems that the nginx proxy that serving the request to app engine flex can't be customized. So no way to update the SSL policy
Comment: The explanation in this answer also confirms it: https://stackoverflow.com/a/47617671/4495081
Comment: @DanCornilescu, so it seems in the app engine load balancer level and there is no way to control it, right?
Answer: So I also came up against this problem...and found that GCP weren't that helpful. They'll helpfully restrict at a domain level if a support ticket is put forwards....which resolves the security concern...but you'll still get false positives which need explaining at every penetration test (the GAE shared IPs accept other version of TLS for other domains).
For a nice clean solution; use Cloudflare for your DNS. They essentially act as a middleman/web application firewall. Amongst other things (free certificates, WAF, DDOS mitigation, CDN, HTTPS force, HSTS etc etc etc), you're able to set the minimum TLS version as you wish. Mine is now minimum TLS 1.2, supporting TLS 1.3 if the browser accepts it. I've also essentially only got port 80/443 on GAE connected to cloudflare, with no public access at all, as all traffic goes through cloudflare first. Pretty neat - zero ports open to the public and a fully operations website! The pen test guys just scratched their heads and packed up.
Oh...and FYI - it's free for this level of configuration. Happy security testing ;-)
Comment: There's now a solution for this situation within GCP, which is [serverless NEGs](https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts). You'll be able to configure your own endpoint without a shared IP and whatever certs you want.
Comment: In 2023, is there a way to set min TLS of 1.2 using GCP *without* the additional cost associated with implementing a static ip and load balancer / forwarding rules? AWS is setting this as standard in their serverless products, so I am surprised Google is still requiring customers to go through this process and incur the additional costs.
Answer: I can confirm that you can make a request to google support and it takes up to 4 weeks to make the change. Not sure why. Hopefully they can speed things up in the future. But alternatively you can handle this logic at the application layer (in middleware) rather than the network layer. See snippet below:

<code>// using NODEJS + TYPESCRIPT
// disable tls 1.0 and 1.1 weak ciphers
this.app.use((req, res, next) => {
      // const cipher = ((req.socket) as TLSSocket).getCipher()
      const protocol = ((req.socket) as TLSSocket).getProtocol()
      // console.log('cipher: ', cipher);
      // output eg: { name: 'ECDHE-RSA-AES128-GCM-SHA256', version: 'TLSv1/SSLv3' }
      console.log('protocol: ', protocol);
      // output eg: TLSv1.2

      if (protocol === 'TLSv1.2' || protocol === 'TLSv1.3') {
          next();
      } else {
          res.status(426);
          res.send('request requires TLSv1.2 or greater, please upgrade');
      }
  });</code>
Answer: I've not tried this so I can't guarantee it would work, but it seems like you could use a HTTP(S) Load Balancer.  The SSL policies are configurable such that it would likely meet the requirements of your security review.

Google App Engine Node.js TLS 1.2


Upgrading Injectivity of a *-homomorphism From a Dense Subalgebra

Question: In the proof of Lemma A.4 of this document, the author proves that a C$^*$-algebra is simple by showing that every one of its representations is faithful. They proceed by taking a representation and show it is injective on a (fixed) dense $^*$-subalgebra. Then they claim that the restriction of the representation to this subalgebra is faithful (obvious) and that the extension back to the original algebra is faithful (valid conclusion?).
Perusing other questions on here it would appear that there is a problem with the final claim, if we just look at extending continuous injective functions then you can't just conclude that the extension is injective, and if we look at extending injective $^*$-homomorphisms on dense $^*$-subalgebras, then there are still problems in the general case.
I suppose that my question would be the following; what extra conditions may we impose on an injective $^*$-homomorphism $\varphi:\mathcal{D}\rightarrow B$ between a dense $^*$-subalgebra $\mathcal{D}$ of a C$^*$-algebra $A$, and another C$^*$-algebra $B$ so that it's extension to $A$ is injective?
The only thing which springs to mind is that if we insist that $\varphi$ is an isometry, then its extension will also be an isometry, hence injective.
Answer: First, as you say it is not true that a faithful representation on a dense subalgebra extends to a faithful representation. For an example, let $A=C[0,1]$, $D$ the polynomials, and let $\pi:D\to \ell^\infty$ be given by
$$
\pi(p)=(p(1),p(1/2),p(1/3),\ldots).
$$
Then $\pi$ is faithful, but its (unique) extension to $C[0,1]$ is not (as there exist many functions $f$ with $f(1/n)=0$ for all $n\in\mathbb N$).
As for a sufficient condition, one possibility is to require $D$ to have the property that for any positive $a\in A$ there exists positive $d\in D$ with $d\leq a$. Then $\pi(a)\geq \pi(d)$, and so $\pi(a)$ cannot be zero for any positive $a\in A$.
Comment: The polynomials aren't dense in $C_0(\mathbb R)$, in fact, the only polynomial in $C_0(\mathbb R)$ is the constant $0$ polynomial.
Comment: But this can be modified to $A=C([0,1])$, $D$ the polynomials, $\pi: D\to\ell^\infty$ by $\pi(p)=(p(1),p(\frac{1}{2}),p(\frac{1}{3}),\ldots)$.
Comment: You are right, I didn't use $[0,1]$ to go for the easy sequence, but it doesn't work. I'll edit.


How to replace and save a word in text file using regular expression in python

Question: Here is my program I want to replace a word to another word in a text file using a regular expression but I am not able to save those words in the text file. Can anyone, please help me to save a file. Thank you in advance.
Given below is my code:
import re
with open("c:\Users\Desktop\hh.txt","r+") as f:
     for i in f.readlines():
        content=re.sub("hai","welcome",i)
        #after replace  how can i save these words in text file again

Answer: A simple approach for small files is to do your reading and writing separately.
import re

path = 'hh.txt'

with open(path, "r") as f:
    oldlines = f.readlines()

newlines = []

for line in oldlines:
    newlines.append(re.sub("hai", "welcome", line))

with open(path, "w") as f:
    f.writelines(newlines)

If you're dealing with huge files, I suggest you write to a temporary file while reading from your input file. Then do a file delete then a file rename.


Easy way to reduce prefab complexity?

Question: Is there a simple way to reduce the complexity of prefabs that I'm using in Unity?  I notice that some prefabs I download are "Low Detail" which render far more quickly on my HoloLens device.  Other high quality prefabs don't do so well.  Is there a "slider" or something similar where I can adjust the quality of the prefab (or its underlying components like materials, textures, etc), without having to do work in Blender (outside of my skill set)?   I know the Unity project Quality Settings are helpful, but those settings impact the entire experience, instead of specific prefabs that are causing problems.  Thanks.
Comment: What you need is different "LOD" ("quality") set of prefabs, there's no workaround for that (except if you create your own meshes and prepare them, but as you mention prefabs, this is a different story I assume. Here's a doc page for a better understand: http://docs.unity3d.com/Manual/LevelOfDetail.html
Comment: No, you must hire a studio to make different assets.  Your question would be much like asking "I saw a movie on the weekend and I want a different movie."
Answer: There is no magic wand to fix objects. You need to learn a little about 3d modelling and Uvmapping.
Fast way to increase your frame rate and reduce render time and memory is doing a clean mesh with the fewest polygons posibble and minor texture size possible. Ever trying to use quads instead of triangles or ngons.
Uvmap texture also they influence in performance. Little textures = little resources to handle.
In the market are several softwares that can help you to reduce your mesh sacrifying quality, and each 3d software how 3d Max , Lightwave 3d, Cinema 4d, blender, etc have their own tools to reduce or clean mesh.
Sometimes is better a new retopology of your object than a reduce polygons algorithm for a clean mesh with low poly count.
Comment: right @JoeBlow, in 3d models is better to use quadrilaterals polygons for smooth edges and easy uvmaping , but later their becom to be triangles. 1 quad = 2 triangles. A little reference: http://blog.digitaltutors.com/modeling-with-quads-or-triangles/
Comment: (Your answer is correct of course but) quads?  what do you mean there?
Comment: @JoeBlow He probably means Quads: https://en.wikipedia.org/wiki/Types_of_mesh#Quadrilateral
Comment: sure... mesh in Unity has only triangles. There is no quad concept in the pipeline. I guess, Jorel means that you can use abstraction tools in Maya etc., en route to your final lower poly models for use in Unity.  (Which can only be ordinary everyday triangle mesh.) In any event this is the correct answer - the first two sentences!
Comment: Hi Jorel.  Yes, we build mesh generation engines.  I'm not sure if you've ever used Unity3D as such.  There are **no** quads in the pipeline.  (Note that for the record, there are much, much more efficient heuristics than just converting each abstract quad to two triangles.)  Just as I said in the previous comment, I assume you mean use "quads" in Maya or the like, and let it reduce to tris for you for use in Unity. (Of course, pro low-poly artistes work at the triangle level for ultimate efficiency.)
Comment: @JoeBlow i understand you, Unity3d is a bad place to use how 3d model tool. Really the right explanation is to use 2 coplanars triangles, but i think is more easy to understand to use a quadrilateral polygon. Unity3d only work with triangle and all 3d software finally on render time convert all mesh to triangles internally. But if you add a simple 3d primitive box in Unity3d you can see that there is 6 quads...6x2 coplanar trianlges :)

Easy way to reduce prefab complexity?


Return id of row that has duplicate data

Question: I am needing to get the row id of rows which have duplicate
Select Name from table1 group by Name having count(1) > 1

table1
ID | Name    | ClientID
----------------------------
01 | John    |  01
02 | Sam     |  01
03 | Sue     |  01
04 | John    |  02
05 | John    |  01

the only problem is that it will only return the Name and not the ID of the column as I have multiple clients on the same table and I don't want to count the names from the other clients as duplicates.
So is there a way to find duplicate data in a table and return the ids of the rows so that I can then use those id's in another query?
Answer I went with modified
Thank you every one for the answers I went with this one which is a modification of the one I marked as the answer.
select t1.*
from (select count(*) over (partition by entityname) as cnt ,t1.*
      from table1 t1 where ClientID = 1
     ) t1
where cnt > 1 and ClientID = 1 order by cnt;

Answer: Use a window function:
select t1.*
from (select t1.*, count(*) over (partition by name) as cnt
      from table1 t1
     ) t1
where cnt > 1;

The count(*) over (partition by name) counts the number of rows for each name.  However, it does this by appending the count on each row, not by reducing the number of rows.  That's the information you need for selecting the rows.
Comment: Wow. ok so can you explain it for me? this looks like what I am wanting to use but I can't quite understand what is going on there. Thank you.
Answer: Select Name, min(ID) ROWID From table1 Group BY Name Having Count(ID)>1

min(ID) here will return the first time the ID appears in the duplicate, and the Count(ID)> 1 will filter out the rows where you have duplicates.
Good Luck!
Comment: So this will give me one id of every duplicate?
Comment: It would tell you the first time a Name appears on the list.  The query will return one line for each name, and the row id column will contain the ID where the ID appears on the original file.
Comment: Great! That is exactly what I was looking for. Thank you
Answer: To see the link of dupes
Declare @Yourtable table (ID varchar(25),Name varchar(50),Client_ID varchar(25))
Insert into @Yourtable values
('01','John','01'),
('02','Sam' ,'01'),
('03','Sue' ,'01'),
('04','John','02'),
('05','John','01')

Select A.*
      ,B.Dupes
 From  @YourTable A
 Cross Apply (Select Dupes=(Select Stuff((Select Distinct ',' + cast(ID as varchar(25))
                              From @YourTable
                              Where ID<>A.ID and Name=A.Name
                              For XML Path ('')),1,1,'')
                            )
              ) B
 Where Dupes is not null

Returns
ID  Name    Client_ID   Dupes
01  John    01          04,05
04  John    02          01,05
05  John    01          01,04

Answer: You can query like this
;WITH cte_duplicates
AS (SELECT
  id, name, client_id,
  ROW_NUMBER() OVER (PARTITION BY name ORDER BY id) AS rc
FROM @Yourtable)
SELECT
  id, name, client_id
FROM cte_duplicates
WHERE rc > 1

Answer: If you wanted to fileter out the duplicates based on both the name and ClientId, use the below query.
 ; with cte_1
   as (select *, count(*) over (partition by name,client_id order by ID) as dups
  from table1  )
 Select *
 From cte_1
 where dups> 1;


Computer settings in group policy are not applying...User settings applied easily..how?

Question: We have a in house domain controller (DC1.xyz.local) & child domain controller (cdc1.abc.xyz.local)...I have below mentioned problems -

None of the computer settings getting applied on to any of the computers.Tried lots of things on computer side.Checked network setting 'they are perfect'..wherein user settings are applied.Is it something related to forest-child domain controller issue? or am I having a faulty desktop image? or a DNS server issue? Need urgent help!!!!
I dont see Default domain policy in child domain controller's GPMC. Is it because it is a child domain, or someone has deleted it? If yes, what shall I do?
Answer: I have seen the exact symptom described here, user settings applying but not computer settings, in exactly two cases:

The computer was in a remote office and required a VPN to connect to the enterprise network.
In the wireless network that required a captive-portal login to connect to the enterprise network.

In both cases, the computer wasn't able to log in to the domain-controllers before the user login. Because of that, they weren't able to pull the GPOs needed. Once the user logged into either the VPN or the captive-portal settings were able to be applied but not consistently. This hit imaged machines worst, as they frequently would get thrown on the requires-login network so wouldn't be able to cache the GPOs for application at boot. The fix was one of two things:

Have freshly imaged machines log in via the wired network once before sending out to wireless.
Poke holes in the captive portal to allow AD login w/o portal login.


Click the close button will close all bootstrap model

Question: For those code, there have a button "Launch modal" to open the Model, then in Model, also have a button "Launch modal2" to open a new model(just call model2) on Model. When I click the close button on model2, all of the Model will de dismiss. How can I fix that?
Please note that I need to put model2 in Model for display data.
Thanks.
http://jsfiddle.net/ve42xs6w/
Launch modal

          <button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button>

Modal title

          Content for the dialog / modal goes here.

          Launch modal2

                    <button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button>

Second Modal title

                    Content for the dialog / modal goes here.

                    Close
                    Save changes

          Close
          Save changes

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.0.0/css/bootstrap.min.css" />

Answer: The problem is that you are putting the second modal inside of the first modal. Bootstrap just closes all modals that the close button is inside of, so if you nest them the second close button will close all outside modals.
To fix this you can just move the whole div#myModal2 down under the first div#myModal so they are separate elements.
Like this: http://jsfiddle.net/e92o1j3s/3/


PDF view not presenting

Question: I'm downloading a PDF from a url and then displaying it using PDFKit in Swift5. However, when I tap my viewCell it is not presenting the PDF and potentially not downloading the PDF. How can I verify this?
I have two files, PlanogramViewController and ShowPDFViewController.
var pdfURL: URL!

func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
        if let plano = self.data[indexPath.row] as? Planogram {
            guard let url = URL(string: Plano.pdfUrl) else { return }

            let urlSession = URLSession(configuration: .default, delegate: self, delegateQueue: OperationQueue())

            let downloadTask = urlSession.downloadTask(with: url)
            downloadTask.resume()

            let pdfViewController = ShowPDFViewController()
            pdfViewController.pdfURL = self.pdfURL
            present(pdfViewController, animated: false, completion: nil)
            tableView.deselectRow(at: indexPath, animated: true)
        }

    }

extension PlanogramViewController:  URLSessionDownloadDelegate {
    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
        print("downloadLocation:", location)
        // create destination URL with the original pdf name
        guard let url = downloadTask.originalRequest?.url else { return }
        let documentsPath = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0]
        let destinationURL = documentsPath.appendingPathComponent(url.lastPathComponent)
        // delete original copy
        try? FileManager.default.removeItem(at: destinationURL)
        // copy from temp to Document
        do {
            try FileManager.default.copyItem(at: location, to: destinationURL)
            self.pdfURL = destinationURL
        } catch let error {
            print("Copy Error: \(error.localizedDescription)")
        }
    }
}

So I'm not getting any download verification from my extension. It should be printing my initial download location and then moving it to the cache. If it encounters an error it should also print an error, but I'm getting nothing in the console. As if nothing is downloading.
Then here is the pdfViewController it's supposed to present once it has the PDF downloaded:
import UIKit
import PDFKit

class ShowPDFViewController: UIViewController {

    var pdfView = PDFView()
    var pdfURL: URL!

    override func viewDidLoad() {
        super.viewDidLoad()

        view.addSubview(pdfView)

        if let document = PDFDocument(url: pdfURL) {
            pdfView.document = document
        }

        DispatchQueue.main.asyncAfter(deadline: .now()+3) {
            self.dismiss(animated: true, completion: nil)
        }
    }

    override func viewDidLayoutSubviews() {
        pdfView.frame = view.frame
    }
}

The current behavior is it just highlights the cell I tapped. Nothing in the console and nothing presented on the display.
Answer: You need to conform to the UITableViewDelegate and UITableViewDataSourceDelegate protocols in the view controller that has the tableView.
and add
tableView.delegate = self
tableView.dataSource = self

in the viewDidLoad()
Then, implement the required methods, and add optional methods to further customize  your tableView.
Comment: I have already conformed in my code (I just didn't include the class declaration in the post, my bad)

PDF view not presenting


How to forbid hide content of the html table column when column size is auto?

Question: I have an HTML page and its width is limited. Also, a table on that page so the width of the table limited by the width of the page. Then I have several columns in that table the width of the column is calculated automatically. If the width of the column's content (let's say these are text strings without breaks) is bigger than the width of the page then part of the strings will be hidden. I want to forbid hide the content for some particular column. In other words, I want that column takes as much width as it needs and the other columns split the rest of the space among them.
How can I do it?
Answer:
a block-level element always takes up the full width available
  (stretches out to the left and right as far as it can).
Setting the width of a block-level element will prevent it from
  stretching out to the edges of its container. Then, you can set the
  margins to auto, to horizontally center the element within its
  container. The element will take up the specified width, and the
  remaining space will be split equally between the two margins:

source: https://www.w3schools.com/css/css_max-width.asp

div.ex1 {
    width:500px;
    margin: auto;
    border: 3px solid #73AD21;
}

div.ex2 {
    max-width:500px;
    margin: auto;
    border: 3px solid #73AD21;
}
<!DOCTYPE html>

This div element has width: 500px;

This div element has max-width: 500px;

Tip: Drag the browser window to smaller than 500px wide, to see the difference between
the two divs!