How can I solve error gypgyp ERR!ERR! find VSfind VS msvs_version not set from command line or npm config?

Question: I want to run this project : https://github.com/adonis-china/adonis-adminify
When I run npm install, there exist error :
> email@example.com install C:\laragon\www\adonis-admin\node_modules\sqlite3
> node-pre-gyp install --fallback-to-build

node-pre-gyp ERR! Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v64-win32-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for firstname.lastname@example.com and email@example.com (node-v64 ABI) (falling back to source compile with node-gyp)
node-pre-gyp ERR! Tried to download(undefined): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v64-win32-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for email@example.com and email@example.com (node-v64 ABI) (falling back to source compile with node-gyp)
gyp ERR! gypfind VS
 gyp ERR!ERR!  find VSfind VS
 msvs_version not set from command line or npm config
gypgyp  ERR!ERR!  find VSfind VS msvs_version not set from command line or npm config
 VCINSTALLDIR not set, not running in VS Command Prompt
gyp gypERR! ERR!  find VSfind VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp checking VS2019 (16.2.29230.47) found at:
 gypERR!  find VSERR! checking VS2019 (16.2.29230.47) found at:
gyp  find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional"
ERR!gyp find VS ERR! "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional"
 gypfind VS ERR! - "Visual Studio C++ core features" missing
gyp  ERR!find VS  - "Visual Studio C++ core features" missing
find VSgyp could not find a version of Visual Studio 2017 or newer to use
 gypERR!  ERR!find VS  could not find a version of Visual Studio 2017 or newer to use
find VS looking for Visual Studio 2015
gyp gyp ERR!ERR!  find VSfind VS looking for Visual Studio 2015
 - not found
gyp gyp ERR!ERR!  find VSfind VS - not found
 not looking for VS2013 as it is only supported up to Node.js 8
gyp ERR!gyp  ERR!find VS find VS not looking for VS2013 as it is only supported up to Node.js 8

gyp gypERR!  ERR!find VS
 gypfind VS  **************************************************************
gypERR!  ERR!find VS find VS **************************************************************
 You need to install the latest version of Visual Studio
gypgyp ERR!  ERR!find VS find VS You need to install the latest version of Visual Studio
 including the "Desktop development with C++" workload.
gypgyp  ERR!ERR! find VS find VS including the "Desktop development with C++" workload.
 For more information consult the documentation at:
gyp ERR!gyp find VS  For more information consult the documentation at:
ERR!gyp  ERR! find VSfind VS https://github.com/nodejs/node-gyp#on-windows
 https://github.com/nodejs/node-gyp#on-windows
gyp gyp ERR!ERR! find VS  **************************************************************
find VSgyp **************************************************************
 gypERR! find VS
ERR! find VS
gyp gypERR!  ERR!configure error
 configure errorgyp
 ERR! stackgyp Error: Could not find any Visual Studio installation to use
 gypERR!  stackERR! Error: Could not find any Visual Studio installation to use
 stackgyp      at VisualStudioFinder.fail (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:121:47)
ERR!gyp stack      at VisualStudioFinder.fail (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:121:47)
ERR! gypstack      at findVisualStudio2013 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:74:16)
gypERR!  ERR!stack      at findVisualStudio2013 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:74:16)
stack     at VisualStudioFinder.findVisualStudio2013 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:344:14)
gypgyp ERR! stack     at findVisualStudio2015 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:70:14)
 gypERR!  ERR! stackstack     at regSearchKeys (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:365:16)
     at VisualStudioFinder.findVisualStudio2013 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:344:14)
gyp gypERR!  ERR!stack stack     at regGetValue (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\util.js:54:7)
     at findVisualStudio2015 (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:70:14)
gypgyp  ERR!ERR!  stackstack     at C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\util.js:33:16
gyp     at regSearchKeys (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:365:16)
 gyp ERR! ERR!stack     at ChildProcess.exithandler (child_process.js:301:5)
 gypstack      at regGetValue (C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\util.js:54:7)
ERR! gypstack     at ChildProcess.emit (events.js:182:13)
 gypERR! ERR!  stack     at C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\util.js:33:16
stack     at maybeClose (internal/child_process.js:962:16)
gyp gypERR! ERR!  System Windows_NT 10.0.17134
stackgyp     at ChildProcess.exithandler (child_process.js:301:5)
 ERR!gyp  commandERR! "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Chelsea\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "--fallback-to-build" "--module=C:\\laragon\\www\\adonis-admin\\node_modules\\sqlite3\\lib\\binding\\node-v64-win32-x64\\node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=C:\\laragon\\www\\adonis-admin\\node_modules\\sqlite3\\lib\\binding\\node-v64-win32-x64" "--python=C:\\Users\\Chelsea\\.windows-build-tools\\python27\\python.exe"
 gypstack     at ChildProcess.emit (events.js:182:13)
 gypERR!  ERR!cwd C:\laragon\www\adonis-admin\node_modules\sqlite3
 gypstack ERR!     at maybeClose (internal/child_process.js:962:16)
 node -v v10.15.0
gypgyp  ERR!ERR!  Systemnode-gyp -v Windows_NT 10.0.17134
 v5.0.3
gypgyp  ERR!ERR! command  "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Chelsea\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "--fallback-to-build" "--module=C:\\laragon\\www\\adonis-admin\\node_modules\\sqlite3\\lib\\binding\\node-v64-win32-x64\\node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=C:\\laragon\\www\\adonis-admin\\node_modules\\sqlite3\\lib\\binding\\node-v64-win32-x64" "--python=C:\\Users\\Chelsea\\.windows-build-tools\\python27\\python.exe"
not okgyp
 ERR! cwd C:\laragon\www\adonis-admin\node_modules\sqlite3
gyp ERR! node -v v10.15.0
gyp ERR! node-gyp -v v5.0.3
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --module=C:\laragon\www\adonis-admin\node_modules\sqlite3\lib\binding\node-v64-win32-x64\node_sqlite3.node --module_name=node_sqlite3 --module_path=C:\laragon\www\adonis-admin\node_modules\sqlite3\lib\binding\node-v64-win32-x64 --python=C:\Users\Chelsea\.windows-build-tools\python27\python.exe' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\laragon\www\adonis-admin\node_modules\sqlite3\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
node-pre-gyp ERR! System Windows_NT 10.0.17134
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\laragon\\www\\adonis-admin\\node_modules\\sqlite3\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\laragon\www\adonis-admin\node_modules\sqlite3
node-pre-gyp ERR! node -v v10.15.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.38
node-pre-gyp ERR! not ok
Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\Chelsea\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --module=C:\laragon\www\adonis-admin\node_modules\sqlite3\lib\binding\node-v64-win32-x64\node_sqlite3.node --module_name=node_sqlite3 --module_path=C:\laragon\www\adonis-admin\node_modules\sqlite3\lib\binding\node-v64-win32-x64 --python=C:\Users\Chelsea\.windows-build-tools\python27\python.exe' (1)
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: firstname.lastname@example.com (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for email@example.com: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! email@example.com install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the firstname.lastname@example.com install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Chelsea\AppData\Roaming\npm-cache\_logs\2019-09-10T22_53_41_072Z-debug.log

How can I solve the error?
Comment: At first, I would run Visual Studio Installer and add "Visual Studio C++ core features", because it says it wants it and it is missing in your VS2019 installation.
Comment: An obsolete `node-sass` version was the cause for me, fixed using an answer in [npm install with error: `gyp` failed with exit code: 1](https://stackoverflow.com/questions/49348482/npm-install-with-error-gyp-failed-with-exit-code-1/70896127#70896127)
Comment: windows-build-tools is corrupting my anaconda installation by installing a separate python 2.7. For my problem, updating npm version from 6 -> 8, resolved the issue without asking for any python at all. So, **I would suggest updating npm** before installing windows-build-tools
Answer: TL;DR
Use the Visual Studio Installer to get the Desktop development with C++ workload in one of the Visual Studio versions you have installed in your machine:

2024 update: also "check the Windows 10 SDK in the installation details" (thanks Nicholas)
Reason/Details
Reading through the log, the main error is due to this:

msvs_version not set from command line or npm config

After this one you find a few of these:

"Visual Studio C++ core features" missing

And later:

You need to install the latest version of Visual Studio including the "Desktop development with C++" workload.

For more information consult the documentation at:

VS https://github.com/nodejs/node-gyp#on-windows

Finally:

Could not find any Visual Studio installation to use

So to solve the problem, you just need to get "Desktop development with C++" workload.
If you have a Visual Studio version installed

open Visual Studio Installer (Win + search for it)
on the list displayed with all Visual Studio Installations you have in your PC, press the Modify button of one of them (preferably the latest version)
on the Workloads grid/list select the checkbox with Desktop development with C++ (2024 update: also "check the Windows 10 SDK in the installation details" (thanks Nicholas))
Press one of the Install buttons

gyp will then find that version and use it:

gyp info find Python using Python version 3.8.1 found at "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\python.exe"

gyp info find VS using VS2019 (16.4.29709.97) found at:

gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\

With NO Visual Studio installed
The following solutions assume you have node installed. Disclaimer: I did not test any of them, but I appreciate everyone that reported them solving this issue.
Alternative 0: before installing additional software
As kaulshapranav wrote in a couple of comments, slightly shortened/rephrased:

windows-build-tools installation log mentions this tools package is deprecated and is already included with node. It corrupts my anaconda installation by installing a separate python 2.7 (...); updating npm version from 6 -> 8 solved the issue.

npm comes with node, so the solution: update your node/npm version. However, if you need to retain specific versions, nvm (notice the v) allows to change between them: simple tutorial on how to use nvm.
Alternative 1
As Hamid Jolany's answer suggests, on an admin shell, simply install the build tools package globally (node-gyp README):
npm i -g windows-build-tools

Alternative 2
Ragavan's idea/answer from a similar thread:

verify if you have VS build tools by running npm config get msvs_version (if you have, skip to step 4 and attempt setting the environment variables)
if not installed, get the VS build tools latest version exe and install it (Ragavan's idea was with or Microsoft Build Tools 2015)
run npm config set msvs_version 2019 --global (or npm config set msvs_version 2015 --global according to Ragavan's idea)
Possibly optional: set the VCTargetsPathenvironment variable (Win + search for var) to the appropriate path (e.g. C:\Program Files (x86)\MSBuild\Microsoft\Portable\v5.0) or through Run as Admin terminal, as in Ragavan's idea with the 2015 Build Tools:

set VCTargetsPath="C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140"
or in Powershell: $env:VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140

Side notes:

a previous error might be resolved by installing Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019;
you might also want to restart after installing any of the above
Comment: C++ workload, `Total space required 6.66 GB`, if that isn't a bad sign, i don't know what is
Comment: It is 7.5 GB disk space to install the visual c++ stuff
Comment: Alternative 1 seems simple but pollutes python spaces (I have anaconda installed with 3.9 and windows-build-tools installs python 2.7). Also when we do `npm i -g windows-build-tools`, the first line it shows on the console is that that package is deprecated and the build tools are already included with node.
Comment: I had only Visual Studio 2022 with "Desktop Development with C++" installed and had this issue. I had to install "Build Tools for Visual Studio 2019" from https://my.visualstudio.com/Downloads?q=visual%20studio%202019&wt.mc_id=o~msft~vscom~older-downloads which fixed it.
Comment: `npm config set msvs_version "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise"` was required for me on top of the answer, but definitely worked. Thank you!
Comment: I think node-gyp still requires python2.
Comment: `choco install nodejs-lts python2 visualstudio2017buildtools visualstudio2017-workload-vctools -y` I believe I was able to fix this exact issue with this `choco` command.
Comment: windows-build-tools is corrupting my anaconda installation by installing a separate python 2.7. For my problem, updating npm version from 6 -> 8, resolved the issue without asking for any python at all. So, I would suggest updating npm before installing windows-build-tools
Comment: I aslo had to check the Windows 10 SDK in the installation details, it was not working with Windows 10 SDK
Comment: With the 2019 version, I found this error: `ValueError: invalid mode: 'rU' while trying to load binding.gyp`.
Comment: don't forget to restart terminal after install the "Desktop Development with C++"
Comment: Adding the C++ Desktop workload in VS 2022 doesn't solve the problem: still keeps complaining about missing `Microsoft.VisualStudio.VC.MSBuild.Base`
Answer: Install all the required tools and configurations using Microsoft's windows-build-tools using npm install --global windows-build-tools from an elevated PowerShell or CMD.exe (run as Administrator).
Full Guidance in GitHub
Comment: can you check afterwards which version it effectively installed ?
Comment: This tool (windows-build-tools) is now deprecated, node should do this for you now.
Comment: @PeterS Node still doesn't do it well at all, it's not being detected at all
Comment: @sw1337 perhaps create a new question and refer to it here? Include any set-up you may use. This is now working for me. To continue to use the windows-build-tools, isn't advised: https://www.npmjs.com/package/windows-build-tools
Comment: @PeterS I ended up making it work. My problem was Node/Python left junk behind even after uninstall and that was messing up the process. These days Node includes all that you need if you pick the "Install additional tools" option in the installer. However, your system has to be squeaky clean before you attempt to do that. That means completely deleting all instances of Node, Python and manually deleting garbage left over such as dirs and env variables (including in the path). Disable your firewall and then run the install. It should download and install everything and work after a reboot.
Comment: @sw1337 Good tips and that makes sense, my system was a new VM, so no old installs etc. Well done on fixing.
Answer: To get this to work with Visual Studio 2022, I had to run:
npm config set msbuild_path "C:\Program Files\Microsoft Visual Studio\2022\Community\Msbuild\Current\Bin\MSBuild.exe"

Comment: In my case it worked after downgrading node.js to v16.13.0
Comment: I installed Desktop Development with C++ workload and I tried this but still I get the same error. Do I need to install windows build tools?
Comment: No, well at least I didn't have to do that, another thing I did to fix this error when it appeared again on an older project was downgrading node.js to version 14.16.1
Comment: In my case it was the other way around - on upgrading from v16.17.0 to v18.17.0 (the latest at the time of writing this), this issue was resolved
Answer: I solved the issue with npm install -g node-gyp, then npm install.
Comment: It did the trick for me. Apparently, the `node-gyp` version upgrade helped.
Comment: Couldn't set the msvs_versions, this is the only thing that helped
Answer: Use the following command before npm install:
npm config set msbuild_path "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe

Changing \Enterprise\ to \Community\ worked for me.
Comment: The path for me was `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe`
Comment: for me it was "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\MSBuild.exe"
Answer: I finally solved it on my end
here is how I did it

run this command on visual studio: npm config get msvs_version( mine was saying undefined)
then I run npm config set msvs_version 2022 --global(I was not seeing undefined anymore)
Then run: npm install. then I run my automation but this time I was seeing (Miscellaneous Warning SELF_SIGNED_CERT_IN_CHAIN:)
then I run set NODE_TLS_REJECT_UNAUTHORIZED=0 and voila!!! it is working

I hope this helps .it was really annoying because some of my automation file was not working
Comment: I am getting "msvs_version` is not a valid npm option" when trying to do this.
Comment: @pooya13 I had the same error. Just run `npm config edit` and add all your key values pair (msvs_version=2022 or msbuild_path=...)
Answer: I wanted to give a recent update. After dealing with this problem for some time, it has become pretty clear to me that some versions of node (currently using 14.19.1) are not able to properly detect build tools for Visual Studio 2022. However, they have no problem detecting build tools for Visual Studio 2019. I was able to get everything working by having Visual Studio 2019 installed alongside Visual Studio 2022. First, you have to undo all the fixing you've been trying to do:
npm config delete msvs_version
npm config delete msvs_version --global
npm config delete msbuild_path
npm config delete msbuild_path --global

Then, download Visual Studio 2019 Community, and during install you must select "Desktop development with C++". At this point, when running npm install, node-gyp will still find your Visual Studio 2022 installation directory and attempt to use it (even if you set msvs_version to 2019), so you must add VCINSTALLDIR into your User environment variables (Start, System Properties, Environment Variables) using this directory:
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC

After this, run "refreshenv" at the command prompt, then run npm install, and feel refreshed that the suffering is over. These kind of problems are very unfortunate for us programmers. Hope this works for you.
Comment: Thank you so much. I had tried all the suggestions I could find on internet, but only this one works for me.
Comment: This is the most current and working answer for 2023-12-08
Answer: Visual Studio Installer -> Modify Installed(Visual Studio Community 2019) -> Workloads(tab) -> [x] Desktop development with C++ -> Install
Answer: Start by running this (you can edit the year and type of VS you have i.e 2022 and community)
npm config set msbuild_path "C:\Program Files (x86)\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe"
Afterwards Run npm install -g node-gyp, then npm install.
Make sure all this is done within the project.
Answer: Try
npm install --global --production windows-build-tools --vs2015

It worked for me.
Answer: A nice alternative it is to reinstall nodejs add ensure that the checkbox in "Tools for Native modules" step it is CHECKED. This is very important because it will install all the things needed to run apps in this kind of scenarios. I have face the same issue and this solve it.
nodejs "Tools step"
After the nodejs installation will start the installation the other tools in power shell, in my case in a few moments the process stops but restart with only pression enter. After a few minutes all was done
Answer: I tried to solve this issue using npm i -g windows-build-tools but not working then I install Visual Studio Build Tools desktop development with c++ tools
and finally successfully solved this issue
Answer: I only installed C++ build tools (refer picture) through Visual Studio Build Tools (2019 version). No other steps required for me on Win 10 Pro.
Answer: I got the same issue and was confused about what I should do. I searched the whole web and still no solution.
What I decided to do is that
npm outdated

When you try this command you will see all the versions basically and in detail you understand which version is needed, which is the latest version at that moment, and which version your project json file has.

C:\GitHub\portfolio>npm outdated
Package                          Current  Wanted  Latest  Location                                      Depended by
babel-plugin-styled-components    1.13.3  1.13.3   2.0.7  node_modules/babel-plugin-styled-components   portfolio
babel-preset-gatsby               1.14.0  1.14.0  2.21.0  node_modules/babel-preset-gatsby              portfolio
eslint                            7.32.0  7.32.0  8.22.0  node_modules/eslint                           portfolio
gatsby                            3.14.6  3.14.6  4.21.1  node_modules/gatsby                           portfolio
gatsby-plugin-google-analytics    3.14.0  3.14.0  4.21.0  node_modules/gatsby-plugin-google-analytics   portfolio
gatsby-plugin-image               1.14.2  1.14.2  2.21.0  node_modules/gatsby-plugin-image              portfolio
gatsby-plugin-netlify             2.11.1  2.11.1   5.0.1  node_modules/gatsby-plugin-netlify            portfolio
gatsby-plugin-offline             3.10.2  3.10.2  5.21.0  node_modules/gatsby-plugin-offline            portfolio
gatsby-plugin-react-helmet        3.10.0  3.10.0  5.21.0  node_modules/gatsby-plugin-react-helmet       portfolio
gatsby-plugin-sitemap             2.12.0  2.12.0  5.21.0  node_modules/gatsby-plugin-sitemap            portfolio
gatsby-plugin-styled-components   3.10.0  3.10.0  5.21.0  node_modules/gatsby-plugin-styled-components  portfolio
gatsby-remark-prismjs             3.13.0  3.13.0  6.21.0  node_modules/gatsby-remark-prismjs            portfolio
gatsby-transformer-remark         4.11.0  4.11.0  5.21.0  node_modules/gatsby-transformer-remark        portfolio
husky                              6.0.0   6.0.0   8.0.1  node_modules/husky                            portfolio
lint-staged                       10.5.4  10.5.4  13.0.3  node_modules/lint-staged                      portfolio
react                             17.0.2  17.0.2  18.2.0  node_modules/react                            portfolio
react-dom                         17.0.2  17.0.2  18.2.0  node_modules/react-dom                        portfolio

Then after I figure out all reds to yellow, all the packages are installed and ready to use.
Now try it again.
Worked for me; it should work for you too.
Answer: My solution was combination of all previous fixes:

Install Visual Studio 2017 (https://my.visualstudio.com/Downloads?q=Visual%20Studio%202017 - It will ask you to log in, choose Build Tools for Visual Studio 2017 (version 15.9))
When installing, choose Desktop development with C++ (or similar name)

In C:\Users\<your_username>\ open .npmrc file and delete everything inside, save and close file

Now you need to set environment variables. If you don't know how to do it google it.
3.1 In section user variables click new, set variable name as VCINSTALLDIR and variable value as C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\ (note that path can be different in your case but make sure that VC path is provided).

In terminal do:
4.1 npm config set msbuild_path "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe"
(note that location can be different in your case, but make sure you are specifying correct path to MSBuild.exe including MSBuild.exe)
4.2 npm config set msvs_version 2017

Now you can run build or install. When I did all the steps, I didn't get error.
Comment: for step 1 just `npm config edit`
Answer: Cool!
Following old answers, npm i -g windows-build-tools is an easy and successful answer. But windows-build-tools exists a bug with issue: https://github.com/felixrieseberg/windows-build-tools/issues/244. Try and do it!
Answer: just update module gulp-sass to the latest version, add sass module to the package.json and write
const sass = require('gulp-sass')(require('sass'));

to the gulpfile.js to get rid off node-gyp.
Answer: i fixed by install vscode 2017 and Desktop development with C++(im install with default optinal select download. not sure if not select default optinal see image.
will error or not) with vsInstaller and set (please set with Developer Command Prompt for VS 2017) use command with your directory project not global example: cd "G:\work\projectA"

set msbuild_path by command:
npm config set msbuild_path "G:\vs 2017\MSBuild\15.0\Bin\MSBuild.exe" (im setup with drive G by initiate because my c drive is full)
set msvs_version by command:
npm config set msvs_version 2017 (before im use 2015 can't work)
im check node-gyp can work? by command
node-gyp configure if not err. congat you can use npm install now

note: you can check local value by npm config get ....
Answer: If all of above doesn't help, verify that your command prompt is not customized
E.g., if you're using MS Windows and you use doskey to add aliases on startup.
The tools that look for Visual Studio installation get mixed up if this happens.
You can test if this is the case by manually opening a new command prompt and verifying it has bare minimum output, such as
Microsoft Windows ...
(c) ...
$Path>
Answer: For me worked the trick of using of npm 14.14.0
gyp started to work perfectly well
ps for my project I also installed Visual Studio Build Tools (2022 version) with Desktop development with C++ workload and Python311 with configuring its PATH in win10
Answer: I had got this issue after deciding to re-install node.js (update to latest version).
I'm on Windows 11 and was installing node.js 18.16.1 (LTS).
There is a checkbox that needs to be selected, to install the additional tools, which I've done. This in turns runs a PowerShell script that will install all that is required. For myself the script finished rally quickly, saying that nothing was done -- I thought that I must have all that's needed since I had Node installed previously.
The problem actually was that it was using chocolatey to install it, and as it was already present under: C:\ProgramData\chocolatey it was skipping the whole thing.
You need to remove the entire folder - C:\ProgramData\chocolatey and run the node.js installer once more.
Then the script will take quite a long time and might seem to be stuck (you can monitor the resource manager -- either CPU, disk or WiFi will be busy)
I have also had some config in the .npmrc file, which I've added as suggested by some answers on here. You can run npm config edit and remove those values -- for myself the file was all commented out then.
Additionally you need to have Python 3.10 on the Path -- it (currently) doesn't work with 3.11 (however this was giving a different error at later stage).
Answer: I had the same problem and spent hours trying to fix this. For me, the problem was the version of node used. The node I used had an internal version of node-gyp that was very old (version 5.1.0) and is NOT compatible with Visual Studio 2022.
There are two ways to resolve this:

Use a more advanced version of node (I tried node 16.0.0 and above and they all worked for me). Versions 14.17.5 and below didn't work for me.
If you can't upgrade your version of node due to dependency issues, then updating your node-gyp globally won't work and you must update it internally within the location where you node version is installed. It is not complicated. Follow the instructions here: https://github.com/nodejs/node-gyp/blob/main/docs/Updating-npm-bundled-node-gyp.md
Note: Using the instructions in this link - I tried installing the latest version of node-gyp and it caused new problems. So I tried using node-gyp version 9.0.0 and that worked for me.
Answer: In my case it will be difficult to resolve because installing programs on my work machine involves authorization from managers.
I discovered that a lib that my project used was causing a conflict, lib 'node-rfc', I simply went back to an older version and the error stopped occurring.
I returned from 3.3.0 to 2.7.0.
and it worked
Answer: I have had that problem with email@example.com on Windows 10 with Visual Studio 2022.
I think v7.1.2 might be not able to work with Visual Studio 2019/2022 properly.
Just try this: find a module in dependencies which uses firstname.lastname@example.com - for me it was gl, for you seems like sqlite3
and add an override below into the main package.json
"overrides": {
    "sqlite3": {
        "node-gyp": "9.3.1"
    }
}

remove node_modules and try npm install again. Hope it helps.
Answer: I updated the component version and ran it successfully, reducing the version of the component. Python 2.7. Nodejs16.20.0. Visual Studio 2019.
The original version was Python 3.10.9 Nodejs18.19. Visual Studio 2022
Answer: Using
npm config set msvs_version=2022 --global
npm ERR! `msvs_version` is not a valid npm option

with space or with = errored for me.
However, npm config edit adding msvs_version={version} in the file directly worked fine.
npm config get msvs_version
2022

Answer: Update for 2024, sqlite3 lists node-gyp:8 in its dependencies. Windows 11 SDK was added in 9. You need to install Windows 10 SDK even if you have 11.
Answer: I was able to fix this exact issue with this choco command
choco install python --version=3.10.0
choco install microsoft-build-tools --version 12.0.21005.20140416
choco install visualstudio2017community
choco install visualstudio2017-workload-nativedesktop
choco install nodejs.install --version=14.21.2

If you re using NodeJS version 14.X, you should go with Visual Studio 2017.
Answer:
If you are still facing issues with installation of windows build
tool. you can also install it manually without using command prompt.

you can install windows build tools through official site or by this also here.
1: Windows Build Tool Complete Package : Here
2: Official site to download: Here
Answer: Downgrade from version 18 to version 16.. it worked for me.
Answer: Ensure that you have the latest npm installed.
npm install -g npm@latest
Answer: I also faced the same issue, what I did is I reinstall the nodejs


Click function that opens and closes

Question: I am trying to code something like this one on http://teamgeek.co.za/#who
I already had the code on the gif and div that will show the description on the bottom of each picture selected. The problem is that if I clicked on the second row, the opened description on the first row won't automatically close itself.
Here is my script that I used.

This is the full code that I used, and I am using the javascript above to get the functions:

Greggy Rick Go

/Chief Executive Officer

Dennise Recuerdo

/Secretary

Jude Marlon Alegro

/Head Web Developer

Hannah Lois Aliposa

/Head Content Writer

Ruth Danielle Aliposa

/Head Web Designer

Christopher Emmanuel Socong

/Web Developer

Angineth Bantiles

/Web Content Writer

Ian Kevin Mendova

/Web Developer

Comment: Why don't you create one click function for all the pictures, hide the text via class selecter's `hide` function and then show the current text?
Comment: add a fiddle that will help.
Answer: Below code optimization of toggle for your issue
you have set class as .row-data for each row and the and write below script..

Note: remove your Script...
Answer: If you want to go with this code you should maybe use this:

I'm not sure if I understand the snippet correctly since I don't know the rest of your code :-)
However, this isn't a good way to solve your issue. As in the comments already mentioned, it's a lot better to handle it with classes.
Here is a jsfiddle which can easily solve your problem:
JSFiddle
However you might only change the effect, but this is a minor thing.
Comment: This fucntion on JSFiddle is the one I am looking for. I am having a bit of trouble on going around with the codes again in applying the codes you have provided.
Comment: I got this one to work. :)

But one problem though, once I click on the image again it wont close.
Comment: did you use toggle? `.toggle("slow");`, if you only use `.show()` it wont toggle :-)
Comment: actually I used the script you pasted on your answer.

Click function that opens and closes


Probability and counting cards

Question: The problem goes like this:
"I am given 7 cards from a regular 52 playing card deck."
"Find the probability that there are at least 3 of the cards equally high (e.g. that there are 3 or more jacks). Put up the answer although it is not necessary to calculate the result. Rationalize your solution to the problem."
I put up a solution that looks like this:
$$\frac{13\binom43\binom{49}4+13\binom44\binom{48}3}{\binom{52}7}$$
Is this the right solution to the problem?
Answer: It's a little more complicated than that. Call a hand of seven cards good if it contains at least $3$ cards of the same value. The problem is that you've counted some good hands twice in your numerator. For instance, the hand with all four aces and all of the kings except the king of clubs is counted once in the term $13\binom43\binom{49}4$ and once again in the term $13\binom44\binom{48}3$; if you replace the ace of spades with the two of spades, the resulting hand is counted twice in the term $13\binom43\binom{49}4$, once for the three aces and once for the three kings.
Try splitting up the possibilities a little more: count separately the hands that have just a three of a kind, the hands that have just a four of a kind, that have two threes of a kind, and that have a three of a kind and a four of a kind.
Comment: I have separated the hands like this:

Hands which have only 3 of a kind:
$13\binom{49}1$

Hands which have only 4 of a kind:
$13\binom{48}1$

Hands which have two threes of a kind:
$13\binom43 12\binom43$

Hands which have a three of a kind and a four of a kind:
$P(13,2)\binom43\binom44$

Are those correct?
Comment: @drleifz: Not quite. Take the first one: there are $13$ ways to pick which rank gets the three of a kind, but then there are $\binom43$ ways to choose which three of the four cards of that rank are chosen. Finally, the remaining four cards can be of four different ranks, of three different ranks with one occurring twice, or of two different ranks, each occurring twice. There are $\binom{12}4$ ways to choose four of the remaining $12$ ranks, and $4^4$ ways to pick one card from each of those ranks, for a total of $4^4\binom{12}4$ ways to pick the last $4$ cards so that they're all of ...
Comment: ... different ranks. If they're of $3$ different ranks, there are $12$ ways to pick the rank that occurs twice, $\binom42$ ways to pick $2$ cards of that rank, $\binom{11}2$ ways to pick the other $2$ ranks, and $4^2$ ways to pick one card of each of those ranks, for a total of $12\cdot4^2\binom42\binom{11}2$. And so on. It's not hard, but it is a bit messy. The easiest case is a three of a kind and a four of a kind: $13$ ways to pick the four of a kind, $12$ ways to pick the rank for the three of a kind, and $4$ ways to pick which card of that rank is *not* included: $13\cdot12\cdot4$.
Comment: Do I need to do this for all of the separated hands?
Comment: @drleifz: You have to make sure that every good hand gets counted exactly once, so yes, each of the categories is going to require some further splitting. The three of a kind category is the worst, though; the others all have fewer possibilities to consider.
Comment: That's true. Sounds like I need a couple of pages for this one :)
Comment: @drleifz: Depends on how big you write. :-) More seriously, what I suggested is the most straightforward approach but not the shortest. If you're comfortable with the inclusion/exclusion idea, you can get a shorter calculation along the lines that Graham Kemp suggests in his answer.
Comment: I think I would get the best understanding by doing it the most straightforward approach.
Comment: Now I have counted all the possibilities (i think..) and that has left me with the following:

Hands that have exactly 3 of a kind:
$13*\binom43$
where the possibilities for the 4 remaining cards are:
$\binom{12}4*4^4$  // This is four different kinds
$12*4^4*\binom42*\binom{11}2$ // This is 3 different kinds (one comes two times.)
$12*11*4^2$          // This is if there are 2 different kinds

Hands that have exactly four of a kind:
$13*\binom44$
where the possibilities for the remaining cards are:
$\binom{12}3*4^3$ // Three different kinds
$12 * 11 * \binom42 * 4$  // Two different kinds.
Comment: Hands with two three of a kind:
$13 * \binom43 * 12 * \binom43 * \binom{46}1$
Hands with a three of a kind and a four of a kind:
$13 * \binom43 * 12 * \binom44$

So now I am just wondering how I put all this together... And is this a correct counting?
Comment: The pieces are almost right. You should have $4^2$, not $4^4$, in the second piece. In the third piece there are $12$ ways to choose the rank that occurs twice, $\binom42$ ways to pick to cards of that rank, $\binom{10}2$ ways to pick the other $2$ ranks, and $4^2$ ways to pick one card from each, so $12\cdot11\cdot4^2$ is definitely wrong. When you have two threes of a kind, the extra card must come from the $44$ cards that aren't of those ranks, not from the $46$ cards that you haven't chosen already. The others are okay. To put them together, you need to add up the possibilities. This ...
Comment: ... easy with the ones in the second half of your comment. In the one three-of-a-kind case you have to add up all of the ways to choose the other four cards and then multiply that by the $13\binom43$ ways to pick the three-of-a-kind. The four-of-a-kind case is similar.
Comment: What I just did is that I calculated all the hands. Meaning that first I calculated how many ways I can draw 7 cards with 3 the same rank and the other 4 of different ranks, the other four of 3 different ranks (1 rank occurring twice), the other four of 2 different ranks (with 2 ranks occurring twice). Then I added the outcome of those and got how many possibilities it is to draw 7 cards with exactly 3 of the same rank. And then I did the same with the other ranks and then I summed all the outcomes together and divided with $\binom{52}7$. How does that sound? It gave me 7,8% probability.
Comment: @drleifz: I think that you're off just a bit: I get a little less than $7,7$%. Part of the problem may be that I made some typos in the comment in which I gave some calculations. Here are the subtotals that I get: $4$ & $3$: $624$; two $3$s: $54.912$; just $4$: $224.224$; just $3$: 10.007.712. Total:10.287.472
Answer:
"Find the probability that there are at least 3 of the cards equally high (e.g. that there are 3 or more jacks). Put up the answer although it is not necessary to calculate the result. Rationalize your solution to the problem."

You have the probability space counted right: ${52\choose 7}$
For the favourable events you need to use the Principle of Inclusion and Exclusion to count ways to get either 3 or 4 cards of any one face, and discount ways to get exactly 3 cards each of two faces, or 4 cards of one face and 3 cards of another.
$$\frac{\overbrace{{13\choose 1}{4\choose 3}{48\choose 4}}^\text{3 suits of 1 face}+\overbrace{{13\choose 1}{4\choose 4}{48\choose 3}}^\text{4 suits of 1 face}-\overbrace{{\Box\choose \Box}{\Box\choose \Box}{\Box\choose \Box}{\Box\choose \Box}}^\text{3 suits each of 2 faces}-\overbrace{{\Box\choose \Box}{\Box\choose \Box}{\Box\choose \Box}{\Box\choose \Box}{\Box\choose \Box}}^\text{3 suits of 1 face 4 suits of another} }{52\choose 7}$$
Comment: Isn't this similar to the method described above? Although this is a little bit more constructive
Answer: Another approach would be to find the probability of the complementary event, and then subtract from 1.
If we don't have 3 cards of the same rank, then there are 4 possibilities:
1) 7 singletons $\;$
2) 1 pair and 5 singletons$\;$
3) 2 pairs and 3 singletons$\;$
4) 3 pairs and 1 singleton.
This gives a probability of
$1-\displaystyle\frac{\binom{13}{7}4^7+\binom{13}{1}\binom{4}{2}\binom{12}{5}4^5+\binom{13}{2}\binom{4}{2}^2\binom{11}{3}4^3+\binom{13}{3}\binom{4}{2}^3\binom{10}{1}4}{\binom{52}{7}}$.

Probability and counting cards


Need help with Boolean alg

Question: There's a Boolean algebra which happens:

For any $y\in B$, if $y+y'=0$ then $y=0$

We need to Refute or prove
Comment: Have you tried trivial Boolean algebras, such as $1$-element or $2$-element Boolean algebras?
Comment: no, I didn't try it
Answer: Since $y+y'=1$, the statement is either vacuously true or $B$ is the trivial one-element Boolean algebra.

Need help with Boolean alg


Sorting lines of text file numerically when the file has strange formatting

Question: I'm having an issue comprehending how to do this. I have a txt file handling birthdays as such:
**January birthdays:**
**17** - !@Mark
**4** - !@Jan
**15** - !@Ralph

**February birthdays:**
**27** - !@Steve
**19** - !@Bill
**29** - !@Bob

The list continues for every month, each month is separated by a blank line. How on Earth do you sort the days sequentially with formatting like this?
For example January should be:
**January birthdays:**
**4** - !@Jan
**15** - !@Ralph
**17** - !@Mark

What I've brainstormed:
I thought maybe I could potentially use readlines() from specific indexes and then save each line to a list, check the integer somehow, and then re-write the file properly. But this seems so tedious and frankly seems like the totally wrong idea.
I also considered using partial() to read until a stop condition such as the line of the next month and then sort somehow based on that.
Does Python offer any easier way to do something like this?
Comment: are you trying to sort the days within each month, or sort both months and days?
Comment: @rudiejd sorry for any confusion! Only the days within the month. The text file is properly formatted for the months, but the days within each month are all over the place.
Comment: Are you in control of how this txt file is created? If so I suggest formatting it in a way that's easier to parse.
Comment: Unfortunately not @Jab, the text file is created as such and I have no access to change it
Answer: You can do it as follows.
Code
import re

def order_month(month_of_entries):
    '''
        Order lines for a Month of entries
    '''
    # Sort key based upon number in line
    # First line in Month does not have a number,
    # so key function returns 0 for it so it stays first
    month_of_entries.sort(key=lambda x: int(p.group(0)) if (p:=re.search('\d+', x)) else 0)

# Process input file
with open('input.txt', 'r') as file:
    results = []
    months_data = []
    for line in file:
        line = line.rstrip()
        if line:
            months_data.append(line)
        else:
            # blank line
            # Order files for this month
            order_month(months_data)
            results.append(months_data)

            # Setup for next month
            months_data = []
    else:
        # Reached end of file
        # Order lines for last month
        if months_data:
            order_entries(months_data)
            results.append(months_data)

# Write to output file
with open('output.txt', 'w') as file:
    for i, months_data in enumerate(results):
        # Looping over each month
        for line in months_data:
            file.write(line + '\n')
        # Add blank line if not last month
        if i < len(results) - 1:
            file.write('\n')

Output
**January birthdays:**
**4** - !@Jan
**15** - !@Ralph
**17** - !@Mark

**February birthdays:**
**19** - !@Bill
**27** - !@Steve
**29** - !@Bob

Alternativee, that also sort months if necessary
import re
from itertools import accumulate
from datetime import date

def find_day(s, pattern=re.compile(r'\d+')):
    return 99 if not s.strip() else int(p.group(0)) if (p:=pattern.search(s)) else 0

def find_month(previous, s, pattern = re.compile(fr"^\*\*({'|'.join(months_of_year)})")):
    ' Index of Month in year (i.e. 1-12)'
    return months_of_year.index(p.group(1)) if (p:=pattern.search(s)) else previous

with open('test.txt') as infile:
    lines = infile.readlines()

months_of_year = [date(2021, i, 1).strftime('%B') for i in range(1, 13)] # Months of year
months = list(accumulate(lines, func = find_month, initial = ''))[1:]   # Create Month for each line
days = (find_day(line) for line in lines)                               # Day for each line

# sort lines based upon it's month and day
result = (x[-1] for x in sorted(zip(months, days, lines), key = lambda x: x[:2]))

with open('output.txt', 'w') as outfile:
    outfile.writelines(result)

Answer: This program runs under Windows or Linux, which have a sort program. It works by reading in each line of the input file and prepending to each line 4 characters, a 2-digit month number and a 2-digit day number (for the blank line between months it uses '99' as the day number so that it follows all the birthdays for the month). It then pipes these modified lines to the sort program and processes the piped output to remove the first 4 characters and to rewrite the file in place, which means you might want to make a backup of the file before running this in case the computer goes down midway in the processing. It shouldn't be too difficult to modify the code to write the output to a separate file.
This technique is used because no assumption is made on the size of the file -- there could be millions of birthdays for a given month. As long as the sort program can handle the input, this program can.
from subprocess import Popen, PIPE
import sys
import re

p = Popen('sort', stdin=PIPE, stdout=PIPE, shell=True, text=True)
month_no = 0
with open('test.txt', 'r+') as f:
    for line in f:
        if " birthdays:**" in line:
            month_no += 1
            p.stdin.write("%02d00" % month_no)
        else:
            m = re.match(r'\*\*(\d+)\*\*', line)
            if m:
                p.stdin.write("%02d%02d" % (month_no, int(m[1])))
            else:
                # blank line?
                p.stdin.write("%02d99" % month_no)
        p.stdin.write(line)
    p.stdin.close()
    f.seek(0, 0) # reposition back to beginning
    for line in p.stdout:
        f.write(line[4:]) # skip over
    f.truncate() # this really shouldn't be necesssary
p.wait()

Answer: Using collections.defaultdict will be very handy here so you don't need to do any checking you can just add data. You basically just read the file keeping the current month in a variable and check if you're on a new month or not then if you're on a new month just update it, if you're on a date get the day and append the string. (this allows for multiple people to have the same birthday.
from collections import defaultdict

data = defaultdict(lambda: defaultdict(list))

with open('filename.txt') as infile:
    month = next(infile).strip()
    for line in infile:
        if not line.strip(): continue
        if line[2].isalpha():
            month = line.strip()
        else:
            data[month][int(line.split('**')[1])].append(line.strip())

This gets you data neatly into a dict as shown below based on your example:
{'**January birthdays:**': {17: ['**17** - !@Mark'], 4: ['**4** - !@Jan'], 15: ['**15** - !@Ralph']},
 '**February birthdays:**': {27: ['**27** - !@Steve'], 19: ['**19** - !@Bill'], 29: ['**29** - !@Bob']}}

From here you just loop back through the data and just sort the dates as you loop and write to file.
with open('filename.txt', 'w') as outfile:
    for month, days in data.items():
        outfile.write(month + '\n')
        for day in sorted(days):
            for day_text in days[day]:
                outfile.write(day_text + '\n')
        outfile.write('\n')


I am unable to build hybris . Getting error in maven repository WroRunTimeException : connect time out

Question: I am setting up a hybris site which has maven jars dependency . I have copied these jars inside C-->User -->.m2 -->repository
When I run hybris ant all build fails due to this error.

Can you please help me understand what is missing as I am new to maven ?
I have checked the repository folder . It has a 'ro' .Screenshot attached.
Answer: You are not supposed to put jars into that directory. Maven is supposed to download them from a remote repository.


Django - have created a pdf file using ReportLab but how do i store it into a FileField record?

Question: In views.py:
        c = canvas.Canvas(str(num)+'.pdf')
        c.setPageSize((page_width, page_height))
        c.drawInlineImage('logo.jpg', margin, page_height - image_height - margin,
                          image_width, image_height)
        c.setFont('Arial', 80)
        text = 'INVOICE'
        c.save()
        tt2 = Invoices_list.objects.all().filter(Number=num)
        tt2.update(document=c)

In models.py:
class Invoices(models.Model):
    Date = models.DateTimeField()
    Name = models.CharField(max_length=100, blank=True, null=True)
    document = models.FileField(upload_to='temp/', blank=True, null=True)

My view creates a record and completes the Date & Name record, i then create the pdf which works fine (can see it in my root directory) but how do i send it straight to the FileField as the tt2.update(document=c) produces a filename of "<reportlab.pdfgen.canvas.Canvas object at 0x000001D51FB442E0> and not 333.pdf (where num=333) although 333.pdf is in the root of my django project. How do i copy it from my root into the FileField record, and next issue is i host using Heroku so not sure if this will cause additional problems.
Thanks in advance
Any
Answer: You will have to import your Model and then create a new object within your view to populate your database (if this is what you mean with send it to the FileField) like so:
Views.py
from your_app.models import Invoices

c = canvas.Canvas(str(num)+'.pdf')
        c.setPageSize((page_width, page_height))
        c.drawInlineImage('logo.jpg', margin, page_height - image_height - margin,
                          image_width, image_height)
        c.setFont('Arial', 80)
        text = 'INVOICE'
        c.save()

obj, created = Invoices.objects.update_or_create(
                                defaults={
                                    'document': c,
                                }
)

Update:
Please include a doc check in your Model like so:

class Invoices(models.Model):
    Date = models.DateTimeField()
    Name = models.CharField(max_length=100, blank=True, null=True)
    document = models.FileField(
        upload_to='temp/',
        blank=True,
        null=True,
        content_types = ['application/pdf'])

Comment: Tried the above but got error: 'Canvas' object has no attribute '_committed'. I've imported my model already with "tt2 = Invoices_list.objects.all().filter(Number=num)" and tried "tt2.document = c" and "tt2.save()" but also get error
Comment: i've also done similar as you suggested "tt2.update(document=c)" and it passes with no errors but when i look at my file the filename is  "reportlab.pdfgen.canvas.Canvas object at 0x000001CDF2F87CA0" (instead of  "333.pdf" where num=333) and doesn't open my file. Appreciate your help
Comment: Edited my post. Also could you share print(c) after you created the pdf.
Comment: print(c) displays


prove that $F$ is $\mu\times\mathcal{L}$ measurable where $F(n,x)=\frac{(2n+1)^2\sin((2n+1)x)}{(n(n+1))^2}$

Question: Let $\mu$ be the counting measure on $\mathbb{N}$ and $\mathcal{L}$ be the Lebesgue measure on $[0,\pi]$. Define the function $F$ on $\mathbb{N}\times\mathcal{L}$ by $F(n,x)=\frac{(2n+1)^2\sin((2n+1)x)}{(n(n+1))^2}$
Prove that $F$ is $\mu\times\mathcal{L}$ measurable. We need to show that for any $\alpha\in\mathbb{R}$ , $F^{-1}(\alpha,\infty)$ is a measurable set i.e it is in the $\sigma$-algebra generated by measurable rectangles. But I cannot determine $F^{-1}(\alpha,\infty)$? Thanks for your help

Edit:
Here is my second attempt: For every $\alpha \in \mathbb{R}$,   $F^{-1}(\alpha,\infty)$ is open in $\mathbb{N}\times [0,\pi]$ i.e it is a Borel measurable set.

Is it also $\mu\times \mathcal{L}$-measurable set?

Wiki says every Borel measurable set is a Lebesgue measurable set. So $F^{-1}(\alpha,\infty)$ is a Lebesgue measurable set.

Does it mean that  $F^{-1}(\alpha,\infty)$ is $\mathcal{L}\times\mathcal{L}$-measurable?
If the answer of 2 is yes, can we conclude that $F^{-1}(\alpha,\infty)$ is $\mu\times\mathcal{L}$-measurable?
Answer: Here is an important thing to keep in mind:
If you want to show that the function
$$
\Bbb{R} \to \Bbb{R}, x \mapsto e^{x^2 \cdot \sin(x)} - \sum_{n=0}^\infty \frac{x^n}{(n!)^2}
$$
is continuous, you will (hopefully) not try to find for every $\varepsilon > 0$ some $\delta > 0$ such that ...
Instead, you will rely on certain closure properties of the class of continuous functions. I.e.

Sums and Products of continuous functions are continuous
Polynomials and (more generally) convergent power series are continuous on the circle of convergence.
Compositions of continuous functions are continuous.
Uniformly convergent sequences of continuous functions yield continuous limits.

Likewise, if you are trying to show that a "nice looking" function is measurable, you will normally not try to show that $f^{-1} ( (\alpha, \infty))$ is measurable for all $\alpha$. Instead, you will rely on certain closure properties of the class of measurable functions.
In this case,

The projections $\pi_1 : \mathbb{N} \times \Bbb{R} \to \Bbb{R}, (n,x) \mapsto n$ and $\pi_2 : \mathbb{N} \times \Bbb{R} \to \Bbb{R}, (n,x) \mapsto x$ are measurable (if you do not know this, show it using the definition (i.e. calculate $\pi_i ^{-1}((\alpha, \infty))$)).
Products and sums of measurable functions are measurable.
Continuous functions are measurable.
Compositions of continuous functions are measurable.
If $f$ is measurable with $f(x) \neq 0$ for all $x$, then $1/f$ is measurable.

These properties (do you know them?) should allow you to show that $F$ is indeed measurable.
EDIT: I forgot to mention one of the most important closure properties of the class of measurable functions (because it was not needed in this case), namely

Pointwise limits of measurable functions are measurable.
Comment: I can find all properties that you mention at my textbook, thank you for your answer
Comment: I tried to give a proof by using your answer. Could you check my answer? Here I didn't use the fact that the projections are continuous. Was your proof different with mine? Thanks.
Answer: Let $F_1(n,x):=\frac{(2n+1)^2}{(n(n+1))^2}$ and $F_2(n,x):=sin((2n+1)x)$. So it is enough to show that $F_1$ and $F_2$ are $\mu\times\mathcal{L}$-measurable. It is clear that $F_1$ is $\mu\times\mathcal{L}$-measurable. For the measurability of $F_2$, we consider the maps $f: [0,\pi]\to \mathbb{R}$ by $f(x)=x$ and $g:\mathbb{N}\to\mathbb{R}$ by $g(n)=2n+1$. Clearly $f$ is $\mathcal{L}$-measurable and $g$ is $\mu$-measurable. By using this fact, the function $h:\mathbb{N}\times[0,\pi]\to\mathbb{R}$ defined by $h(n,x)=g(n)f(x)$ is$ \mu\times\mathcal{L}$-measurable. We know that $sin$ function is continuous, so it is Borel-measurable. So it is $\mathcal{L}$-measurable since every Borel-measurable function is also $\mathcal{L}$-measurable. So the composition $sin\circ h=F_2$ is $\mu\times\mathcal{L}$-measurable.
Comment: You never defined $f$ and $g$. Why is it "clear" that $F_1$ is measurable? The rest seems nice.
Comment: Sorry, I added the definitions of $f$ and $g$. In addition for any Borel set $B$ in $\mathbb{R}$, we have $F_1^{-1}(B)=A\times [0,\pi]$ for some $A\subseteq\mathbb{N}$. Since $\mu$ is the counting measure, all subsets of $\mathbb{N}$ are measurable. So $F_1^{-1}(B)$ is a measurable rectangle.
Comment: Ok, this makes it a fine proof. But I would (maybe) add the argument for $F_1$ in the answer.


What is the best approach for setting startup variables in Matlab that are unaffected by commands such as `clear`?

Question: I have a situation in which a large amount of data output from Matlab programs needs to be stored in a specific location that can vary from machine to machine.  What I'd like to have is a global variable that is set at startup for each machine.  This variable could then be used by the various programs in order to determine the directory to store the output.  Note the approach will need to be platform independent as there a potentially 3 operating systems that will run these scripts.
One approach I considered is using setenv:
setenv('DATDIR',fullfile(path,to,dir));

Then the script/function could query this variable using:
savedir = getenv('DATDIR');

In this case the variable is unaffected by clear inside the Matlab environment.
Another approach I thought to use is simply creating a function (like matlabroot), which is on the path and simply returns a string to which the data directory path is assigned.
function path = datroot()
    path = fullfile(path,to,dir);
end

Finally, to my question: Is there another way to achieve the same result? If Matlab has it's own environment variables, I'd rather use those instead of functions or system-level variables.  Is this possible?
Answer: Use the getpref/setpref/ispref/addpref/rmpref set of functions. According to the documentation:

Preference values are persistent and maintain their values between
  MATLAB sessions. Where they are stored is system dependent.

This is like .ini or Java .properties files, but instead uses  MAT-files to persist the values (which can be of any MATLAB type). On my Windows machine, the file used is stored as:
%APPDATA%\MathWorks\MATLAB\R2010b\matlabprefs.mat

Comment: thanks for this answer.  `addpref` and `getpref` fit the bill precisely.


How to find Mongoose data recursively?

Question: I am newbie in MEANJS and i have a problem i.e, there are collection called employee and have multiple documents with their boss field. Now i want get all employees with their lower level.
For example:-
1) {_id:ObjectId('587dcd3edca5f235f862fdfd'), name:'John'} //he doesn't have boss
2) {_id:ObjectId('587dcd3edca5f235f86dddew'), name: 'Jimmy', 'boss': ObjectId('587dcd3edca5f235f862fdfd')} //john is boss
3) {_id:ObjectId('587dcd3edca5f235f863ew'), name: 'David', 'boss': ObjectId('587dcd3edca5f235f86dddew')} //john,Jimmy are bosses
4) {_id:ObjectId('587dcd3edca5f235f86qwa'), name: 'Dyan', 'boss': ObjectId('587dcd3edca5f235f86dddew')} //john,Jimmy,David are bosses
5) {_id:ObjectId('587dcd3edca5f235f8ew32'), name:'Jack', 'boss': ObjectId('587dcd3edca5f235f862fdfd')} //john is boss
6) {_id:ObjectId('587dcd3edca5f2wsw23rlot'), name: 'Loren', 'boss':ObjectId('587dcd3edca5f235f8ew32')} //john,Jack is boss
If we take
Jonh then output will ['Jimmy','Jack','David','Dyan','Loren']
Jack then output will ['Loren']
Here is my try code:-
getBosses(user._id)
function getBosses(id){
  User.find({boss:id})
      .exec(function(err,users){
        if(err)
           return console.log(err);

        //How handle here 'users' array
       //for something getBosses call recursively

    })
}

Comment: [**`$graphLookup`**](https://docs.mongodb.com/manual/reference/operator/aggregation/graphLookup/#pipe._S_graphLookup) should do the job quite efficiently.
Comment: According to your example same boss for **David** and **Dyan** so how **David** become boss of **Dyan** ?
Comment: thanks @chridam, but it give opposite result. `aggregate([{$graphLookup: {from: "users",startWith: "$rmUserId",connectFromField: "rmUserId", connectToField: "_id", as: "rmHierarchy"}}])`, here **John** has a zero length of `rmHierarchy` array and **Loren** has all user. However i want **John** should have `rmHierarchy ['Jimmy','Jack','David','Dyan','Loren']`
Answer: As far as I understood you need to find all subordinates of that people. I think the best way to do it is using $graphLookup.
db.bosses.insertMany([
  { _id: "587dcd3edca5f235f862fdfd", name: "John" },
  {
    _id: "587dcd3edca5f235f86dddew",
    name: "Jimmy",
    boss: "587dcd3edca5f235f862fdfd",
  },
  {
    _id: "587dcd3edca5f235f863ew",
    name: "David",
    boss: "587dcd3edca5f235f86dddew",
  },
  {
    _id: "587dcd3edca5f235f86qwa",
    name: "Dyan",
    boss: "587dcd3edca5f235f86dddew",
  },
  {
    _id: "587dcd3edca5f235f8ew32",
    name: "Jack",
    boss: "587dcd3edca5f235f862fdfd",
  },
  {
    _id: "587dcd3edca5f2wsw23rlot",
    name: "Loren",
    boss: "587dcd3edca5f235f8ew32",
  },
]);

db.bosses.aggregate([
  {
    $graphLookup: {
      from: "bosses",
      startWith: "$_id",
      connectFromField: "_id",
      connectToField: "boss",
      as: "subordinates",
    },
  },
  {
    $project: {
      _id: false,
      name: true,
      subordinates: {
        $reduce: {
          input: "$subordinates",
          initialValue: "",
          in: { $concat: ["$$value", ", ", "$$this.name"] },
        },
      },
    },
  },
  {
    $project: {
      name: true,
      subordinates: { $substrBytes: ["$subordinates", 2, -1] },
    },
  },
]);

The result of the last one is:
[
  { name: 'John', subordinates: 'David, Dyan, Loren, Jack, Jimmy' },
  { name: 'Jimmy', subordinates: 'David, Dyan' },
  { name: 'David', subordinates: '' },
  { name: 'Dyan', subordinates: '' },
  { name: 'Jack', subordinates: 'Loren' },
  { name: 'Loren', subordinates: '' }
]

The most important thing is $graphLookup stage of the aggregate pipeline. Last two $project stages is just response formatting - return only name and subordinates as string field with comma separated names.
To get data for a specific person you can use $match stage before $graphLookup like that:
db.bosses.aggregate([
  { $match: { name: "John" } },
  {
    $graphLookup: ...

How to find Mongoose data recursively?


Sending Mail in Python - Error code - SMTPDataError: 451, b 4.3.0 Mail server temporarily rejected message

Question: I am able to send mail without issue when I implement the same code very plain without using any functions. But when I try to send mail using a function I am getting mail as my senders account is disabled and it is blocking my mail and account.
Have to implement a code for sending mails after certain filters are met in a function so that it can be used to send mail whenever wanted.
My code snippet:
import smtplib

def setup_mail():
    global smtpObj,sender,receivers,message
    sender = 'email@example.com'
    receivers = ['firstname.lastname@example.com','firstname.lastname@example.com']

    message = """From: From Person <firstname.lastname@example.com>
    To: To Person <email@example.com>
    Subject: 1 MIN candle 15 points alert

    TIME: watch out next 2 minutes
    """

    smtpObj = smtplib.SMTP_SSL('smtp.gmail.com', 465)
    smtpObj.login("firstname.lastname@example.com","samplepassword")
setup_mail()
def sending_mails(smtpObj):
    smtpObj.sendmail(sender, receivers, message)
for i in range(3):
    sending_mails(smtpObj);
print("Successfully sent email")

The error i got,

Traceback (most recent call last): File "C:/Users/Arun/PycharmProjects/Astro/venv/Scripts/ZERODHA_DEV/sendmail.py", line 22, in  sending_mails(smtpObj); File "C:/Users/Arun/PycharmProjects/Astro/venv/Scripts/ZERODHA_DEV/sendmail.py", line 20, in sending_mails smtpObj.sendmail(sender, receivers, message) File "C:\Python\lib\smtplib.py", line 888, in sendmail raise SMTPDataError(code, resp) smtplib.SMTPDataError: (451, b'4.3.0 Mail server temporarily rejected message. s77sm14888153pfc.164 - gsmtp')

However it works with a simple code like,
import smtplib

global smtpObj,sender,receivers,message
sender = 'firstname.lastname@example.com'
receivers = ['email@example.com','email@example.com']

message = """From: From Person <email@example.com>
To: To Person <email@example.com>
Subject: 1 MIN candle 15 points alert

TIME: watch out next 2 minutes
"""

smtpObj = smtplib.SMTP_SSL('smtp.gmail.com', 465)
smtpObj.login("firstname.lastname@example.com","samplepassword")
for i in range(3):
   smtpObj.sendmail(sender, receivers, message)
   print("Successfully sent email")

Comment: Below is the error i got,Traceback (most recent call last):
  File "C:/Users/Arun/PycharmProjects/Astro/venv/Scripts/ZERODHA_DEV/sendmail.py", line 22, in
    sending_mails(smtpObj);
  File "C:/Users/Arun/PycharmProjects/Astro/venv/Scripts/ZERODHA_DEV/sendmail.py", line 20, in sending_mails
    smtpObj.sendmail(sender, receivers, message)
  File "C:\Python\lib\smtplib.py", line 888, in sendmail
    raise SMTPDataError(code, resp)
smtplib.SMTPDataError: (451, b'4.3.0 Mail server temporarily rejected message. s77sm14888153pfc.164 - gsmtp')
Comment: And  i missed to mention, i have already enabled access for third party and captcha in Gmail setting and location as well
Comment: Are you sure you're not just hitting sending limits? Your question doesn't otherwise make much sense, how would the SMTP server know you're using a function
Comment: Yes, no sending limit reached. When i run this program, it disables my account after sending mail once. So it blocks when it tries to send 2nd time. And i found a stackoverflow page where same issue is faced, but no one suggested solution as well,weird
Comment: Normally that kind of a limitation comes with free service. If you want something more professional, you'll typically have to pay for it.
Answer: The immediate problem is that your message is indented; a valid SMTP message contains two literal adjacent newlines to demarcate the headers from the body.
While you can assemble simple email messages from ASCII strings, this is not tenable for real-world modern email messages. You really want to use the email library (or a higher level third-party wrapper or replacement) to handle all the corner cases of email message encapsulation, especially if you are not an expert on email and MIME yourself.
Comment: I'd show example code but this is a common FAQ; refer to some of my other recent answers in the [tag:python] [tag:email] tags for concrete examples.


$C^*$-ideals in tensor products

Question: This post contains several basic questions about ideals in tensor products of $C^*$-algebras, motivated by the comments in this post.
Let $A$ be a $C^*$-algebra, and let $\mathcal{K}$ denote the compact operators on a Hilbert space $H$. I will use the term "ideal" to mean "closed, two-sided ideal".
Let $I$ be a non-trivial ideal of $A$, i.e. $0\neq I\neq A$. Then $I\otimes\mathcal{K}$ is an ideal of $A\otimes\mathcal{K}$.
Question 1: Is $I\otimes\mathcal{K}$ a non-trivial ideal?
Question 2: Is every ideal of $A\otimes\mathcal{K}$ of the form $I\otimes\mathcal{K}$? Why?
Thoughts: For Q1, one could think of the inclusion $I\otimes\mathcal{K}\subseteq A\otimes\mathcal{K}$ as a direct limit of matrix algebra inclusions $M_n(I)\subseteq M_n(A)$. For each $n$, it is certainly true that $M_n(I)$ is neither zero nor all of $M_n(A)$. But I'm not sure why the proper inclusion still holds after taking the limit.
Also, I wonder if this is true more generally:
Question 3: If we replace $\mathcal{K}$ with an arbitrary $C^*$-algebra $B$, then is the analogue of Q1 still true? More precisely, if we take spatial tensor products, is $I\otimes B$ still a non-trivial ideal of $A\otimes B$? Does the answer change if we take other tensor products?
Answer:
It is always non-trivial. Since the ideal $J$ is a closed subspace of $A$, by Hahn-Banach we can find  $\varphi\in A^*$ with $\|\varphi\|=1$, $\varphi|_J=0$ and $\varphi(a)=1$. Fix $k_0\in K(H)$ and fix a state $\psi\in K(H)^*$ with $\psi(k_0)=1$ (we can take $\psi$ to a point state). Now consider an element $\sum_j b_j\otimes k_j\in J\otimes K(H)$. We have
$$
\|a\otimes k_0-\sum_jb_j\otimes k_j\|\geq \Big|(\varphi\otimes \psi)\big(a\otimes k_0-\sum_jb_j\otimes k_j\big)\Big|=\varphi(a)\psi(k_0)=1.
$$
So $a\otimes k_0$ is at distance 1 from every elements in a dense subset of $J\otimes K(H)$, which shows that $a\otimes k_0\not\in J\otimes K(H)$.

I think the answer is yes, but I would have to think how to prove it.

The argument in 1 applies in general, as long as  $J$ is a closed subspace of $A$. The only nontrivial part of the proof is whether $\varphi\otimes \psi$ defines a bounded linear functional on the tensor product. This can be done for states and any C$^*$-tensor norm by Proposition 3.4.7 in Brown-Ozawa. As any bounded linear functional is a linear combination of states (see for instance Theorem 3.3.10 in Murphy), this works for all bounded functionals.

$C^*$-ideals in tensor products


value replace adds value incorrectly to the end of input

Question: I am trying to do a value replace in an input field. It is almost working correctly. The problem is it adds the value to the end of input when I only want it added to the one location
Html:
<input id="boom" value="03/15/0212" />

Jquery: I have hard coded some values to keep from placing all functions
$('#boom').on('keypress', function (e)
{
    var crs = this.selectionStart;
    var from;
    var to;
    var sub;
    _dateSection = 'month';
    var currentObj = this;
    var currentVal = $(this).val();
    var c = String.fromCharCode(e.which);
    var stringSplit = currentVal.split('/');
    switch (_dateSection) {
        case 'month':
            _currentDateValue = stringSplit[0];
            from = 0;
            to = 2;
            sub = currentVal.substring(from, to);
            break;
        case 'day':
            _currentDateValue = stringSplit[1];
            from = 3;
            to = 6;
            sub = currentVal.substring(from, to);
            break;
        case 'year':
            _currentDateValue = stringSplit[2];
            from = 6;
            to = 10;
            sub = currentVal.substring(from, to);
            break;
    };
    var check = false;
    if (!check) {
        $(currentObj).val(function (index, value)
        {
            return value.replace(sub, c);
        });
    }
});

JsFiddle
If you add 3 to the month portion of the input which is currently populated with 03 it replaces is correctly with 3, but it also adds 3 to the end of the year. Making the year, 20123 when the year value should remain 2012.
The final value should be
3/15/2012

not
3/15/20123

Edit:
this code gives the same problem:
$(currentObj).val(currentVal.replace(sub, c));

as does this
var replaceValue = currentVal.replace(sub, c)
$(currentObj).val(replaceValue);

Comment: why putting a function inside a val? this may introduce scope and enclosure problems
Comment: I updated Code and fiddle to not have function inside. still same issue.
Answer: The extra character that is added to the end is added by the input field itself. The keypress event is handled by the input also, which does the normal thing and adds the character to the value.
Just put e.preventDefault(); in your event handler to keep the input from getting the event.

Also, don't use replace to put the character in the string, use substr to get the parts of the string that you want to keep:
value = value.substr(0, from) + c + value.substr(to, c);

If the string is for example 05/05/2012 and you try to replace the day with 4, it will instead replace the first occurace and you get 4/05/2012 instead of 05/4/2012.
Comment: Thanks for the answer, and also thanks for the bonus info about replace and sub string
Answer: this is working for me:-
$('#boom').on('keyup', function (e)
            {
                var crs = this.selectionStart;
                var from;
                var to;
                var sub;
                _dateSection = 'month';
                var currentObj = this;
                var currentVal = $(this).val();
                var c = String.fromCharCode(e.which);
                var stringSplit = currentVal.split('/');
                switch (_dateSection) {
                    case 'month':
                          _currentDateValue = stringSplit[0];
                          sub = _currentDateValue;
                         c=Number(sub)+Number(c);
                         break;
                    case 'day':
                        _currentDateValue = stringSplit[1];
                        from = 3;
                        to = 6;
                        sub = currentVal.substring(from, to);
                        break;
                    case 'year':
                        _currentDateValue = stringSplit[2];
                        from = 6;
                        to = 10;
                        sub = currentVal.substring(from, to);
                        break;
                };
                var check = false;
                if (!check) {
                    $(currentObj).val(function (index, value)
                    {

                        value=value.substring(from,value.length-1);
                        return value.replace(sub, c);
                    });
                }
        });

Comment: I think that replaces the last number as well. It appears that date becomes 3/15/2013 instead of 3/15/2012


Getting last group_id from MySQL in PHP

Question: I want to get last added group_id from MySQL to get new $group_id++ for other group.
I tried
$group_id = ("select max(group_id) from posts");
  $group_id++;

But when I check it with echo
echo $group_id;

the result is : "select max(group_id) from posts"
Comment: You have written a query. But you have to run it. Please check how to run a query and get the result.
Answer: You're assigning a query string to a variable..you're not even running the query...?
You need to run the query and assign it to a result ($group_id_)
There's other ways to do this but this is similar to what you're doing.
$result= mysqli_query($conn, "select max(group_id) as max from posts");
$row = $result->fetch_assoc($result);
$group_id = $row['max'];
$group_id++;

Comment: @FilipDržik try that
Answer: This is because you are not executing an SQL query, you are doing an assignment.
The correct syntax is something like this:
$result = mysqli_query('SELECT * WHERE 1=1');
if (!$result) {
    die('Invalid query: ' . mysql_error());
}

Search how you can execute SQL queries, and if you have trouble, post again.
Edit: Have a look on the link below
https://secure.php.net/manual/en/mysqli.query.php
Comment: Don't use mysql_query. It is deprecated. Use Mysqli or PDO.
Comment: Mysql is deprecated and you should be using mysqli
Comment: @clearshot66 yeah, my mistake... corrected it as soon as I saw that.... small typo
Answer: This is the easiest way to implement it the latest one will have the highest ID so you order them by their id starting from the highest id to the lowest and you just take the first one.
Read about the http://php.net/manual/en/book.mysqli.php it explains everything about querying data from your database using php.
$mysqli = new mysqli("localhost", "my_user", "my_password", "world");

if ($mysqli->connect_errno) {
printf("Connect failed: %s\n", $mysqli->connect_error);
exit();
}
if($result = $mysqli->query("select group_id from posts ORDER BY group_id DESC LIMIT 1")){
// it has returned a result
$group_id = $result->fetch_object();
$group_id++;
}

Getting last group_id from MySQL in PHP


Is an ANOVA applicable for these data?

Question:
I have a data set from 7 groups, with 20 fish in each group. Measurement of a parameter is made on 25 cells from each fish (so each observation in the data-set is completely independent, right?). One of the groups functions as the control group while other 6 are treatment groups. So we have a total of 25*20*7 measurements. This is how the data looks like (a boxplot of all 7 groups is attached):
samples subjects groups response
    1        1      1     4.85
    2        1      1     3.77 ..
    25       1      1     4.71
    26       2      1     4.51 ..
    500      20     1     4.21
    501      1      2     4.11 ..
    3500     20     7     4.19

I wish to run an ANOVA and the expectation is that a couple of groups should differ from the control group in regards to the parameter under observation. Here are a few questions:

Is the following R code appropriate? (It shows there is no significant difference between groups.)
n = 20
k = 25
g = 7
subjects = gl(n,   k, n*k*g)
groups   = gl(g, n*k, n*k*g)

study1 = data.frame(c(1:(n*k*g)), subjects, groups, r11)
colnames(study1) = c("samples", "subjects", "groups", "response")

fit = lm(response~groups + samples*subjects, data=study1) # or aov?
anova(fit)

Analysis of Variance Table

Response: response
                   Df Sum Sq Mean Sq  F value    Pr(>F)
groups              6  846.8 141.134 122.2864 < 2.2e-16 ***
samples             1   13.1  13.055  11.3114 0.0007787 ***
subjects           19  119.5   6.289   5.4493 2.078e-13 ***
samples:subjects   19  149.6   7.872   6.8206 < 2.2e-16 ***
Residuals        3454 3986.4   1.154
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The attached qqplot shows normal residues, however the Shapiro-Wilk test always fails on all groups. (Is my sample size of 3500 too big and problematic?)
shapiro.test(study1$response[study1$groups==1])

data:  study1$response[study1$groups == 1] W = 0.9818, p-value =
6.648e-06

And so does the Levene for equality of variance:
leveneTest(lm(response ~ groups, data=study1))
Levene's Test for Homogeneity of Variance (center = median)
        Df F value    Pr(>F)
group    6   19.37 < 2.2e-16 ***
      3493

Please guide me as to how should I proceed. Should I keep on using ANOVA and disregard the fact that the normality and equality of variance assumptions are being violated? Should I remove the outliers from my data? Should I transform data somehow to be 'more' normal? Should I switch to non parametric or rank based tests? The end goal is to identify groups that differ significantly from the control group.
Answer: The qq-plot does not show normal residuals.  You can see that the points have a shallow, concave-up curve and the ends move outside of the 95% confidence band (especially at the high end).  Your data are more (positively) skewed than we would expect to find in a sample from a true normal distribution.  In addition, your data are not independent since you have multiple samples from each fish and you have not properly accounted for that in your model.  Thus, your data do not meet any of the three standard assumptions for an ANOVA (1. independence, 2. homoscedasticity, 3. normality).
That said, deviations from normality and homoscedasticity are easier to find with higher N, even if they are too trivial to worry about (see: Is normality testing 'essentially useless'?).  Although I cannot tell how far your data are from a true normal, the plots appear to show fairly minor deviations.  The ANOVA is pretty robust to these violations, especially if they are small or if you have a lot of data.  You probably don't have to worry too much about these problems.
On the other hand, violations of independence tend to be more serious.  It appears to me that you attempted to deal with this by having a samples variable with a unique value for each fish*sample combination in your dataset.  This is incorrect on several levels.  First, you need to group the samples within each fish, not have a unique value for each sample.  Second, the variable should be a factor, not a numeric variable.  Third, fitting this as a fixed effect is possible, but a poor strategy that loses power and most likely does not reflect the question you want to ask.
Instead, you need to fit a mixed effects model.  That is, you need to have a factor variable fish.ID with 7x20=140 levels, where each of the 25 samples from a given fish are assigned to the same level of fish.ID.  Then you would fit a random intercept for each level of the fish.ID variable.  Under the assumption that the data are ordered by samples within the fish, the code might be something like this:
library(lme4)
set.seed(9308)

n = 20
k = 25
g = 7
groups   = gl(g, n*k, n*k*g)
r11      = rnorm(g*n*k)
fish.ID  = factor(rep(1:140, each=25))
study1   = data.frame(r11, groups, fish.ID)
colnames(study1) = c("response", "groups", "fish.ID")

lmm.fit = lmer(response~groups+(1|fish.ID), data=study1)
anova(lmm.fit)
# Analysis of Variance Table
#        Df Sum Sq Mean Sq F value
# groups  6 3.2844  0.5474  0.5383

Comment: Gung,

What you suggested makes a lot of sense. May I seek more help in the direction of interpreting the results of lmm, more specifically obtaining a p-value and group identification for those that differ from control.

    `Fixed effects:
                Estimate Std. Error t value
    (Intercept)   5.0971     0.1436   35.50
    groups2       0.7495     0.2030    3.69
    groups3      -0.4539     0.2030   -2.24
    groups4      -0.1903     0.2030   -0.94
    groups5       0.3045     0.2030    1.50
    groups6      -0.8071     0.2030   -3.98
    groups7      -0.5140     0.2030   -2.53`
Comment: @Apo, that is really a different Q--you might start a new thread. FWIW, this is a common Q. The default advice is a likelihood ratio test. That won't help in your situation. If your control group is set as the reference level / intercept, you could use the summary output (if it had p values) as long as you control for the multiple tests & their non-orthogonality (ie use Bonferroni). There is information about how to get p values [here](http://mindingthebrain.blogspot.com/2014/02/three-ways-to-get-parameter-specific-p.html).
Comment: If I have 4 sets of measurements as descried above, 2 from fish head and 2 from body and tail, would the following formula be correct: response~groups+body_part+(1|fish.ID)
Comment: @Apo, that seems reasonable.
Comment: one last question before I close this thread: would a multivariate like following apply: {resp_head,resp_body,resp_tail}~groups+body_part+(1|fish.ID). How would it differ from univariate?
Comment: @Apo, that doesn't make any sense. You have body parts (in different forms) on both sides of the equation. What you had above is better.

Is an ANOVA applicable for these data?


Is there a standard category-theoretic way to express a loop or quasigroup?

Question: The standard way to encode a group as a category is as a "category with one object and all arrows invertible".  All of the arrows are group elements, and composition of arrows is the group operation.
A loop obeys similar axioms to a group, but does not impose associativity.  Inverses need not exist, but a "cancellation property" exists -- given $xy = z$, and any two of $x$, $y$, and $z$, the third is uniquely determined.
Quasigroups need not even have a neutral element.
Given the lack of associativity, arrows under composition do not work to encode loop elements.
Is there a natural way to do this?
Comment: The lack of associativity is precisely what makes loops _not at all like groups._ As you say, arrows under composition can't model elements of a loop, so I don't see how this is a natural question to ask. The category-theoretic formalism is inherently associative.
Comment: @wnoise: you can use category theory to talk about loops in the sense that you can define a category of loops. I just don't see a natural way to express individual loops as categories.
Comment: @wnoise: When people say category theory is a good way to talk about most fields, they mean that in most fields the objects of interest have some sort of morphisms between them defined and that these form a category. I don't think people mean that the objects themselves are best defined in categorical terms.
Comment: *Semigroups* are "categories with one object"; for groups, you must also require every arrow to be invertible.
Comment: I understand that last complaint. In the course of cleaning up rarely used tags I removed the (loops) tag from this site altogether because there was no consistent use of it -- several people started using it for "loops" as in "loop-space" or "fundamental group". Then I decided to remove the (quasi-groups) tag because there is no universally recognized meaning for it (as is the case with so many quasi-things) and since there were only two or three questions tagged such anyway, so I saw little use for it.
Comment: @Qiaochu Yuan: Lots of people claim that category theory is a good way to talk about any math sub-field.  I'm taking that claim seriously.  Yes, nothing as nice as the group construction seems to work, so just *how* unnatural do we need to go?  We could, for instance, try encoding the multiplication group of a loop, but there are groups that do not arise as the multiplication group of any loop, and there are groups that are the multiplication group of loops of different size.
Comment: @Arturo Magidin: thanks, fixed.
Comment: Have you read Postmodern Algebra by Smith and Romanowska?  I'm not saying it will help but it has quasigroups and loops and category theory in it, but I'm not sure if it combines them.
Comment: I'd be interested in an "external" view as well: what coherence properties on the morphisms are required to get a category where the objects are loops.  But I suppose that'd be a separate question.
Comment: @t.b.: why did you eliminate the "quasigroups" and "loops" tags?  It's true that the site probably doesn't need both  -- but they have aspects which do not appear to be covered under the other tags.
Answer: As expressed by Qiaochi Yuan in this and this comment, the way that category theory applies to studying loops and quasigroups is in the form of a category whose objects are loops, resp. quasigroups.
Only a few structures can actually be described as categories having certain special properties (among which sets, groups, partially ordered sets). For a structure to have any chance of being a "special type of category", it is of course necessary that the defining properties for a category are somehow satisfied by the structure in question.
For loops and quasigroups, this is not obvious to say the least, due to the lack of associativity (which is all-important in category theory).
Answer: I know this is an older thread but I've been thinking about basically this question for a few months now
Something I've come up with is that a quasi-group requires certain automorphisms to exist on the categorical product of 3 elements.
An example of this, in the Category of sets the products AxB, CxA and BxC are all isomorphic to subsets of AxBxC which is isomorphic to CxAxB and BxCxA.
These isomorphisms are what allow us to distinguish between a binairy function and it's left and right inverses, and are generalizable withing the context of categorical products (I believe) as the proper morphisms should exist between categorical products of pairs and products of triples.
Once these automorphisms are established, I believe quasi-groups arise naturally from the existence of morphisms between categorical products of pairs and objects.
This is all really sketchy right now, I'm honestly just getting into category theory but I believe the above works?


Proxy server written in Dart returns HTML instead of web page

Question: I'm working on a proxy server written in Dart and while it forwards GET requests fine, in the browser they're rendered as HTML instead of the actual web page.
Here's my code:
<code>import 'package:http/http.dart' as http;
import 'dart:io';

void launchProxy() async {
  var server = await HttpServer.bind('localhost', 8080);
  await server.forEach((HttpRequest request) async {
    // Parse headers
    final headers = <String, String>{};
    request.headers.forEach((name, values) {
      // Clean up request headers
      String headerValue = values.toString();
      headerValue = headerValue.replaceAll('[', '');
      headerValue = headerValue.replaceAll(']', '');
      // Add the header to the full set of headers
      headers.putIfAbsent(name, () => headerValue);
    });
    //headers.putIfAbsent('Access-Control-Allow-Origin', () => '*');
    // Handle GET request
    if (request.method == 'GET') {
      final response = await http.get(request.uri, headers: headers);
      request.response.write(response.body);
      request.response.close();
    }
    // Handle POST
    else if (request.method == 'POST') {
      // Get the body from the io.HttpRequest
      final bodyBytes = await request.first;
      final response =
          await http.post(request.uri, headers: headers, body: bodyBytes);
      request.response.write(response.body);
      request.response.close();
    }
    // Handle PUT
    // Handle HEAD
    // Handle OPTIONS
    // Handle TRACE
    // Handle DELETE
  });
}
</code>
And here's a screenshot of the view I'm seeing in my browser:
browser screenshot
I've tried a number of different things and this is the furthest I can get with it so now I'm asking for help. I'm not a software engineer by trade, just tinkering for my own ammusement.
Comment: Why aren't you just using shelf, including the shelf_proxy plugin?  Or is it just for your own challenge?
Comment: shelf_proxy, from what I can see, only supports GET requests. I never saw a way to add a request body to it. Also, I need to be able to proxy multiple different requests and from what I can tell, shelf_proxy only does one proxy request to whatever URL you pass it when first created. I'd have to tear down the proxy and re-create it on every new request then.
Comment: What you want isn't a proxy then.  How would you be communicating to indicate how to proxy each request, if not within that request?  Also, it looks like shelf_proxy works with all kinds of http requests, not just GET.
Comment: "in the browser they're rendered as HTML instead of the actual web page." Uh, a web page *is* HTML. Looking at your screenshot, your problem is that the HTML is treated as plaintext. You're copying just the HTTP response body but are ignoring the response's headers. One of those headers is [`Content-Type`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type), which tells the browser whether to interpret the response body as raw text or whether to render it as HTML.
Comment: Well seeing as I'm not in grade school I am aware that web pages are HTML (among other things). I haven't found a way to pass the headers all the way through to tell the browser how to render that which seems to be the problem.
Comment: You can't do `request.responseHeaders['Content-Type'] = 'text/html';`? (Or really you'd want to iterate over the entries in `response.headers` and copy each one.)
Comment: And that was really close to the solution:

`request.response.headers.contentType = ContentType.html;`

I think I'll have to write a function to extract the original content type and apply it.
Answer: And the solution:
<code>// Handle GET request
    if (request.method == 'GET') {
      final response = await http.get(request.uri, headers: headers);
      // This renders the page correctly
      request.response.headers.contentType = ContentType.parse(response.headers['content-type']!);
      // Need to write a function to iterate over all headers and transfer them
      request.response.write(utf8.decode(response.bodyBytes));
      request.response.close();
    }
</code>
I think I'll have to write a function to extract the original content type and apply it.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).


Show that if $f$ is onto then $f(x)=g(x)$ for some $x\in X$.

Question: I'm trying to prove the following:

Let $X$ be a connected space and let $f; g : X \to [0, 1]$ be continuous functions. Prove that if $f$ is onto, then $f(x) = g(x)$ for some $x \in X$.

So I'm saying the following:
Let $E$ be a dense subset of $X$. For any a in $X$ let $\{x_n\}$ be a sequence in $E$ converging to $a$. Then:
$$f(a)=\lim f(x_n)=\lim g(x_n)=g(a)$$
hence
$f(x)=g(x)$ for some $x \in X$
Can I be sure that such a dense $E$ exists?
Comment: Pretty much every response has just told you that your approach is wrong and not suggested anything better. The problem is that you asked a specific question that doesn't really make sense as part of a larger question that *does* make sense. I've gone ahead and edited the post and nominated it for a reopen. The correct approach is as such: if $f(x)\neq g(x)$ for all $x$, then let $A=\{x:f(x)>g(x)\}$, and $B=\{x:f(x)<g(x)\}$. Both are nonempty because $f$ is onto, and they union to all of $X$. Show that both are open (using continuity) to contradict the connectedness of $X$.
Comment: Is the space at least Hausdorff? If not how are you defining convergence to a point? Are you at all interested in the answer with the generalized notion of "dense" rather than the sequential one?
Comment: I'm not sure what this question is asking. Take $X$ to be connected (and nonempty), choose the subset to be the whole space, and choose the sequence to be the constant sequence $x_n = x$ for some $x\in X$. Then $x\in X$ is a limit of $\{x_n\}$.
Comment: Regardless of what may or may not be true for $X$ (which is irrelevant in the scheme of your work), your attempt of proof is circular. It seems to me that you want to say that $f(a)=g(a)$ by picking a sequence $x_n$ of points such that $x_n\to a$ and $g(x_n)=f(x_n)$ for all $n$. But then, what would you need $a$ for?
Comment: I think it is important to notice that this reflects a non-trivial property of $[0,1]$ (so it is kind of a question in algebraic topology although as shown in the comment by @AlexMathers it admits a simple solution in terms of general topology). For example this is not true if you replace $[0,1]$ by the unit circle $S^1$ (take $f$ to be the identity and $g$ a non-trivial rotation).
Answer: The question is alright, but what you are "saying" doesn't make any sense.
It's very simple: By assumption there are $a$ and $b\in X$ with $f(a)=0$, $f(b)=1$. If $g(a)=0$ or $g(b)=1$ we are done. Otherwise the continuous function $h(x):=f(x)-g(x)$ satisfies $h(a)<0$, $h(b)>0$. If $h$ would have no zero on $X$ the union of the two nonempty, open, and disjoint sets
$$\Omega_1:=\{x\in X\>|\>h(x)<0\}\ ,\qquad \Omega_2:=\{x\in X\>|\>h(x)>0\}\ $$
would be $X$, contrary to the assumption on $X$.
Answer: Every topological space is a dense subset of itself (connectedness is unnecessary). And you can use a constant sequence which necessarily converges. Do be careful about convergence in general topological spaces, though, uniqueness of the limit is not guaranteed unless it's Hausdorff.