Managing Memory Leaks in ContentProvider

Question: I have a ContentProvider class, which is utilized by my activities to access, write to, query and delete data from my local database. I am opening my database constantly within the ContentProvider's methods in order to access the data. My question is, where is the proper place to close the database within the ContentProvider?
I tried doing it within the query, insert and delete override methods but keep getting the following Exception:
Caused by: java.lang.IllegalStateException: Cannot perform this operation because the connection pool has been closed.

Here is my ContentProvider Class followed by my Database Class:
public class GroceryContentProvider extends ContentProvider {
final static int FOOD_ITEM = 100;
final static int FOOD_ITEM_WITH_ID = 101;
private UriMatcher uriMatcher = buildUriMatcher();
private final String TAG = "GROCERY_PROVIDER";

public static UriMatcher buildUriMatcher(){
    UriMatcher uriMatcher = new UriMatcher(UriMatcher.NO_MATCH);
    uriMatcher.addURI(FoodContract.CONTENT_AUTHORITY,FoodContract.PATH,FOOD_ITEM);
    uriMatcher.addURI(FoodContract.CONTENT_AUTHORITY,FoodContract.PATH + "/#",FOOD_ITEM_WITH_ID);
    return uriMatcher;
}

@Override
public boolean onCreate() {
    return false;
}

@Nullable
@Override
public Cursor query(Uri uri, String[] projection, String selection, String[] selectionArgs, String sortOrder) {
    Cursor cursor;
    FoodDatabase foodDatabase = new FoodDatabase(getContext());
    int match = uriMatcher.match(uri);

    switch (match){
        case FOOD_ITEM:{
            foodDatabase.openReadableDB();
            cursor = foodDatabase.getAllRows(projection,selection,selectionArgs,sortOrder);

            break;
        }
        default:
            throw new UnsupportedOperationException("Unknown URI: " + uri);
    }
    cursor.setNotificationUri(getContext().getContentResolver(),uri);

    return cursor;
}

@Nullable
@Override
public String getType(Uri uri) {
    return null;
}

@Nullable
@Override
public Uri insert(Uri uri, ContentValues values) {
    int match = uriMatcher.match(uri);
    Uri returnUri;
    switch (match){
        case FOOD_ITEM:{
            Context context = getContext();
            FoodDatabase foodDatabase = new FoodDatabase(context);
            foodDatabase.openWriteableDB();
            long id = foodDatabase.insertRow(values);
            if(id>0){
                returnUri = ContentUris.withAppendedId(FoodContract.FoodList.CONTENT_URI,id);
            }else{
                throw new android.database.SQLException("Failed to insert row into " + uri);
            }
            break;
        }
        default:
            throw new UnsupportedOperationException("Unknown URI: " + uri);
    }
    getContext().getContentResolver().notifyChange(uri,null);
    return returnUri;
}

@Override
public int delete(Uri uri, String selection, String[] selectionArgs) {
    int match = uriMatcher.match(uri);
    int taskDeleted;
    switch (match){
        case FOOD_ITEM_WITH_ID:{
            String idForDeletion = uri.getPathSegments().get(1);
            FoodDatabase foodDatabase = new FoodDatabase(getContext());
            foodDatabase.openWriteableDB();
            taskDeleted = foodDatabase.deleteRow(FoodContract.FoodList.TABLE_NAME,"_id=?",new String[]{idForDeletion});
            break;
        }
        default:
            throw new UnsupportedOperationException("URI UNKNOWN " + uri);
    }
    if(taskDeleted!=0){
        getContext().getContentResolver().notifyChange(uri,null);
    }

    return taskDeleted;
}

@Override
public int update(Uri uri, ContentValues values, String selection, String[] selectionArgs) {
    return 0;
}}

Database:
public class FoodDatabase {
private SQLiteDatabase db;
private DatabaseHelper databaseHelper;
private Context context;
public static final int DB_VERSION = 1;
public static final String DB_NAME = "food_list_db";

private final String CREATE_DATABASE = "CREATE TABLE " + FoodContract.FoodList.TABLE_NAME + " ("
        + FoodContract.FoodList._ID + " INTEGER PRIMARY KEY AUTOINCREMENT, "
        + FoodContract.FoodList.ITEM_NAME + " TEXT NOT NULL"
        + ");";

public FoodDatabase(Context context){
    this.context = context;
    databaseHelper = new DatabaseHelper(context);
}

public FoodDatabase openWriteableDB(){
    db = databaseHelper.getWritableDatabase();
    return this;
}

public FoodDatabase openReadableDB(){
    db = databaseHelper.getReadableDatabase();
    return this;
}

public long insertRow(ContentValues cv){
    return db.insert(FoodContract.FoodList.TABLE_NAME,null,cv);
}

public Cursor getAllRows(String[] projection, String selection, String[] selectionArgs, String sortOrder){
    return db.query(FoodContract.FoodList.TABLE_NAME,projection,selection,selectionArgs,null,null,sortOrder);
}

public int deleteRow(String table, String whereClause, String[]whereArgs){
    return db.delete(table,whereClause,whereArgs);
}

public void closeDB(){
    db.close();
}

private class DatabaseHelper extends SQLiteOpenHelper {
    public DatabaseHelper(Context context) {
        super(context, DB_NAME, null, DB_VERSION);
    }

    @Override
    public void onCreate(SQLiteDatabase db) {
        db.execSQL(CREATE_DATABASE);

    }

    @Override
    public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
        db.execSQL("DROP TABLE IF EXISTS " + FoodContract.FoodList.TABLE_NAME);
        onCreate(db);
    }
}}

Comment: https://groups.google.com/forum/#!msg/android-developers/NwDRpHUXt0U/jIam4Q8-cqQJ
Answer: I suggest you to change the onCreate method of your content provider initializing the FoodDatabase class in a member variable mFoodDatabase:
@Override
public boolean onCreate() {
    mFoodDatabase = new FoodDatabase(getContext());
    return true;
}

And, inside your content provider methods (query, insert, update and delete), instead of re-instantiating the FoodDatabase object, try to use the one stored into the member variable mFoodDatabase.
Hope can help


App crashes when running code from in fragment

Question: I am making my first app in android studio. So far I have built a bottom navigation bar using fragments. I am now trying to add a calendar feature to one of the fragments, I have copied a tutorial using activities to run the code and think there is a problem with using "View".
This is the code to open for the nav bar to open the fragment.
 @Override
        public boolean onNavigationItemSelected(MenuItem item) {
            switch (item.getItemId()){
                case R.id.home:
                    getSupportFragmentManager().beginTransaction().replace(R.id.container,homeFragment).commit();
                    return true;
                case R.id.calendar:
                    getSupportFragmentManager().beginTransaction().replace(R.id.container,calendarFragment).commit();
                    return true;
                case R.id.wellbeing:
                    getSupportFragmentManager().beginTransaction().replace(R.id.container,wellbeingFragment).commit();
                    return true;
            }

            return false;

This is java in my calendar fragment, which when opened from the navigation bar crashes the app
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                         Bundle savedInstanceState) {
    // Inflate the layout for this fragment
    initWidgets();
    selectedDate = LocalDate.now();
    setMonthView();

    return inflater.inflate(R.layout.fragment_calendar, container, false);

}

private void initWidgets()
{
    calendarRecyclerView = requireView().findViewById(R.id.calendarRecyclerView);
    monthYearText = requireView().findViewById(R.id.monthYearTV);
}

private void setMonthView()
{
    monthYearText.setText(monthYearFromDate(selectedDate));
    ArrayList<String> daysInMonth = daysInMonthArray(selectedDate);

    CalendarAdapter calendarAdapter = new CalendarAdapter(daysInMonth, this);
    RecyclerView.LayoutManager layoutManager = new GridLayoutManager(getContext(),7);
    calendarRecyclerView.setLayoutManager(layoutManager);
    calendarRecyclerView.setAdapter(calendarAdapter);
}

This is the error message I receive in my logcat
2022-05-12 12:45:05.305 10950-10950/com.example.myapplication E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.myapplication, PID: 10950
    java.lang.IllegalStateException: Fragment CalendarFragment{c4de180} (94b1c782-ca39-403f-b45b-5d0d7a042f15 id=0x7f080087) did not return a View from onCreateView() or this was called before onCreateView().
        at androidx.fragment.app.Fragment.requireView(Fragment.java:1964)
        at com.example.myapplication.CalendarFragment.initWidgets(CalendarFragment.java:41)
        at com.example.myapplication.CalendarFragment.onCreateView(CalendarFragment.java:30)
        at androidx.fragment.app.Fragment.performCreateView(Fragment.java:2963)
        at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:518)
        at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282)
        at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189)
        at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2100)
        at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002)
        at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.app.ActivityThread.main(ActivityThread.java:7842)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)

I believe there is no error in the calendar code and is something to do with onCreateView in the fragment code.
Answer: You cannot call requireView() before returning a non-null value from onCreateView().
Move these lines
initWidgets();
selectedDate = LocalDate.now();
setMonthView();

(where initWidgets() calls requireView()) from onCreateView() to e.g. onViewCreated().
Answer:
Yes it crashes because findViewById wasn't called from the fragment_calendar layout before returning inflater therefore anything called after the return statement will not be executed and will return a NullPointerException. Unless you override onViewCreated method and put down the codes

So to avoid that, you need to override onViewCreated method or you must return view. Remember, if you want to return view then all codes or public methods must be reached before the return view statement. Else, it'll crash due to exception
Check this codes, it'll fix it correctly
public class TestFragment extends Fragment {
RecyclerView calendarRecyclerView;
TextView monthYearText;
@Nullable
@org.jetbrains.annotations.Nullable
@Override
public View onCreateView(@NonNull LayoutInflater inflater, @Nullable @org.jetbrains.annotations.Nullable ViewGroup container, @Nullable @org.jetbrains.annotations.Nullable Bundle savedInstanceState) {
    selectedDate = LocalDate.now();
    return inflater.inflate(R.layout.fragment_calendar, container, false);
}

@Override
public void onViewCreated(@NonNull View view, @Nullable @org.jetbrains.annotations.Nullable Bundle savedInstanceState) {
    super.onViewCreated(view, savedInstanceState);
    initWidgets(view);
    setMonthView();
}

private void initWidgets(View view) {
    calendarRecyclerView = view.findViewById(R.id.calendarRecyclerView);
    monthYearText = view.findViewById(R.id.monthYearTV);
}

private void setMonthView() {
    monthYearText.setText(monthYearFromDate(selectedDate));
    ArrayList<String> daysInMonth = daysInMonthArray(selectedDate);

    CalendarAdapter calendarAdapter = new CalendarAdapter(daysInMonth, getActivity());
    RecyclerView.LayoutManager layoutManager = new GridLayoutManager(getActivity(), 7);
    calendarRecyclerView.setLayoutManager(layoutManager);
    calendarAdapter.setAdapter(calendarAdapter);
  }
}


Show fields empty via render()

Question: I have a content type with some fields. My client want that the empty fields must to be visible, obviously only the label. Like
Status: New
Owner:
Read: 714

I write these "fields" in my node--XYZ.tpl.php file  via
<? echo render($content['field_all_status']); ?>

But there are not "emtpy fields"
How I can set Drupal to display all fields, empty too?
Answer: Try the Empty fields module. I didn't use it, but look like that's the module you need.

This module provides a way to show empty fields that would otherwise be hidden.

Show fields empty via render()


Will uninstall/reinstall of VMware Tools disrupt a production VM with an E1000 NIC?

Question: A software vendor is urging us to change from the E1000 vNIC to a VMXNET3 on our primary app server. Our CEO has asked me to do it ASAP, so over lunch in about an hour here. I have added the VMXNET3 vNIC to the VM, but the drivers are not loading and so I need to reinstall VMware Tools (it's already up-to-date).
I know it requires a reboot which I will do in my maintenance window over lunch, but does simply reinstalling VMware Tools disrupt a VM that only uses a E1000 vNIC? Thanks.
Answer: Reinstalling VMware Tools should not 'disrupt' a VM using only an E1000 vNIC (it shouldn't change the MAC address or hardware slot ID).
Comment: Thank you, Pak. I went ahead with the procedure and everything went fine, although a reboot of the VM was necessary to install VMware Tools and load the VMXNET3 driver.


Query Xdocument

Question: I have a soap message from an web service as an Xdocument and wish to extract the session id element for future use. I am able to loop over the the response using the code below,  but how would I query the soap to return the sessionid
Dim dnas = From node In Responce.DescendantNodes _
       Select node

        For Each node In dnas
            If TypeOf node Is XElement Then
                Debug.Print(DirectCast(node, XElement).Name.ToString)
            Else
                Debug.Print(node.ToString)
            End If
        Next

The soap:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"     xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<CreateResponse xmlns="http://myweb.com/webservices">
<CreateResult>
    <ResultCode>0</ResultCode>
    <ResultDescription>Authentication Successful</ResultDescription>
    <sessionid>04e1b723-ff93-4af0-94f4-2245ccf11f42</sessionid>
  </CreateResult>
</CreateResponse>
</soap:Body>
</soap:Envelope>

Answer: XNamespace ws = "http://myweb.com/webservices";
string sessionId = (string)Response.Descendants(ws + "sessionId").FirstOrDefault();

Comment: Thanks Martin, that helps a lot.

Query Xdocument


Escolher um índice de uma lista

Question: Tenho uma lista:
l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

Gostaria de escolher um valor entre os índices 6 e 14 dessa lista, mas fazer isso 5 vezes e armazenar esses valores em outra lista. Como faço?
Se fosse de qualquer índice, seria:
random.choose(l)

Mas não é isso que quero. Podem auxiliar-me por favor?
Comment: na verdade `l` é uma lista de listas, algo como: `l = [lista1, lista2, lista3, lista4, ..., lista15]` e `lista1 = [1, 2, 3], lista2 = [4, 5, 6], etc`
Answer: Agora não consigo testar mas creio que pode fazer:
index1 = 6
index2 = 14
vals = []

for i in range(0, 5):
    # l[index1:index2] = lista dos valores entre index1 e index2
    vals.append(random.choice(l[index1:index2])) # aqui armazena os valores

Para não armazenar repetidos em vez do ciclo for faça:
while len(vals) < 5:
    randVal = random.choice(l[index1:index2])
    if randVal not in vals:
        vals.append(randVal)

Cuidado que desta maneira tem de garantir que existem mais de 5 (ou 5) valores entre index1 e index2, senão entra em ciclo infinito.
Comment: joia amigo @Miguel, irei testar! onde estou agora só consigo comunicar por aqui haha
Comment: tem como eu escolher no lugar de um valor do índice, escolher o próprio índice? Fiz aqui e funcionou, mas ele não percorre pelo índice, mas pelo valor de cada item em cada indice.
Answer: É só passar um subconjunto da lista para o random.choice:
random.choice(l[6:14])

Escolher um índice de uma lista


Difficulty running tests, --group option not being respected

Question: I am trying to run a BrowserTestBase based test in my own module, following the documentation on drupal.org.
No matter if I run the test from PHPStorm or command line, I get the same error:
vagrant@local:/var/www/mysite/docroot$ ../vendor/bin/phpunit -c core/phpunit.xml --group mymodule
PHP Fatal error:  Trait 'Drupal\Tests\facets\Functional\BlockTestTrait' not found in /var/www/mysite/docroot/modules/contrib/search_api_solr/tests/src/Functional/FacetsTest.php on line 20
PHP Stack trace:
PHP   1. {main}() /var/www/mysite/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /var/www/mysite/vendor/phpunit/phpunit/phpunit:52
PHP   3. PHPUnit_TextUI_Command->run() /var/www/mysite/vendor/phpunit/phpunit/src/TextUI/Command.php:100
PHP   4. PHPUnit_TextUI_Command->handleArguments() /var/www/mysite/vendor/phpunit/phpunit/src/TextUI/Command.php:111
PHP   5. PHPUnit_Util_Configuration->getTestSuiteConfiguration() /var/www/mysite/vendor/phpunit/phpunit/src/TextUI/Command.php:649
PHP   6. PHPUnit_Util_Configuration->getTestSuite() /var/www/mysite/vendor/phpunit/phpunit/src/Util/Configuration.php:878
PHP   7. PHPUnit_Framework_TestSuite->addTestFile() /var/www/mysite/vendor/phpunit/phpunit/src/Util/Configuration.php:1000
PHP   8. ReflectionMethod->invoke() /var/www/mysite/vendor/phpunit/phpunit/src/Framework/TestSuite.php:353
PHP   9. Drupal\Tests\TestSuites\FunctionalTestSuite::suite() /var/www/mysite/vendor/phpunit/phpunit/src/Framework/TestSuite.php:353
PHP  10. Drupal\Tests\TestSuites\TestSuiteBase->addTestsBySuiteNamespace() /var/www/mysite/docroot/core/tests/TestSuites/FunctionalTestSuite.php:22
PHP  11. PHPUnit_Framework_TestSuite->addTestFiles() /var/www/mysite/docroot/core/tests/TestSuites/TestSuiteBase.php:55
PHP  12. PHPUnit_Framework_TestSuite->addTestFile() /var/www/mysite/vendor/phpunit/phpunit/src/Framework/TestSuite.php:384
PHP  13. PHPUnit_Util_Fileloader::checkAndLoad() /var/www/mysite/vendor/phpunit/phpunit/src/Framework/TestSuite.php:310
PHP  14. PHPUnit_Util_Fileloader::load() /var/www/mysite/vendor/phpunit/phpunit/src/Util/Fileloader.php:38
PHP  15. include_once() /var/www/mysite/vendor/phpunit/phpunit/src/Util/Fileloader.php:56

Fatal error: Trait 'Drupal\Tests\facets\Functional\BlockTestTrait' not found in /var/www/mysite/docroot/modules/contrib/search_api_solr/tests/src/Functional/FacetsTest.php on line 20

I am not requiring Search API Solr in my test, nor do I have a clue why it is running it. Even if I specify --testsuite unit, I get the same error.
Here is the test I am trying to run, with a basic test step:
namespace Drupal\Tests\mymodule\Functional;

use Drupal\Tests\BrowserTestBase;

/**
 * Class AuthenticationTest.
 *
 * @group mymodule
 */
class AuthenticationTest extends BrowserTestBase {

  /**
   * Required modules.
   *
   * @var array
   */
  protected static $modules = ['user', 'node'];

  public function setUp() {
    parent::setUp();

    // set config for netforum
  }

  /**
   * The superuser should not authenticate to Netforum.
   */
  public function testAdminUserUsesDrupalAuthentication() {
    $account = $this->drupalCreateUser(['administer site configuration']);
    $this->drupalLogin($account);
  }

}

I am not sure why it is even looking at the Search API Solr module, but the test in question does try to use several files that do not exist:
use Drupal\Tests\facets\Functional\BlockTestTrait;
use Drupal\Tests\facets\Functional\ExampleContentTrait;
use Drupal\Tests\facets\Functional\TestHelperTrait;

I assume these are a part of the Facets module, which I do not have.
edit: This might be my issue.
Answer:
edit: This might be my issue.

Yes.
The error happens during test discovery, first it has to find all tests and then limit it based on the group.
The easiest workaround is simply to specifiy your module, then you don't even need the group. phpunit will only discover test in that folder, which will not just avoid this problem it will also be much faster.
$ ../vendor/bin/phpunit -c core/phpunit.xml modules/custom/mymodule

Comment: Thanks, lifesaver. The patch in the issue there does not work, and I cannot just dl 8.4.x for `@requires`, and the required modules are listed in require-dev there... messy.
Comment: One note - PHPStorm complains about the directory path since I am running a VM, even though I am using remote interpreter and have path mappings correct. Running from the CLI works in this one instance, though that module should fix this because it blocks a lot of folks from testing or attempting to learn to do functional tests.


Is it possible to use a normal copula after the Johnson transformation?

Question: I have a sample, log-returns of a finance time series. The sample size is n=252. I'd like to estimate a distribution in order to modeling dependence with a copula.
The binned histogram of frequencies, cumulative frequency counts, and qq-plot are below:

I have applied the normality hypothesis test (Anderson-Darling test) and the null hypothesis was rejected at level 0.05 (p-value is 0.003905).
I have applied the Kolmogorov-Smirnov test and the null hypothesis can't rejected at level 0.05 (p-value is 0.21769). But the null hypothesis should be rejected based on the Shapiro-Wilk test (p-value is 0.211e-4), the Chen-Shapiro test (5% critical value is 0.00356).
Original data is here CFC.csv
mydata = read.csv("CFC.csv")  # read csv file

log_returns <- diff(log(mydata$Close), lag=1)
n <- length(log_returns)
mydata$log_ret = with(mydata, c(NA, diff(log(mydata$Close))))
#range(log_returns)
par(mfrow=c(1,3))
hist(log_returns, freq = TRUE, label=TRUE, breaks = 14, xlim=c(-0.08,0.08))
curve(dnorm(x, mean=mean(log_returns),
               sd=sd(log_returns)), add=TRUE, col="darkblue", lwd=2)

Fn = ecdf(log_returns)
plot(Fn,
   main="CDF of log_returns",
   xlab="log_returns",
   ylab="Cumulative Frequency")
qqnorm(log_returns)
qqline(log_returns)

shapiro.test(log_returns)
# Shapiro-Wilk normality test

# data:  log_returns
# W = 0.97509, p-value = 0.000211

# required
library(nortest)
ad.test(log_returns)
# Anderson-Darling normality test

# data:  log_returns
# A = 1.1991, p-value = 0.003905

Problem. Data is not normal. Add I don't know how to specify a distribution to modeling dependence with a copula.
My attempt is: I have applied the Johnson's transformation to normalize the data.
# require
library(Johnson)
#Applying Johnson transformation
log_returns_JT<-RE.Johnson(log_returns)

#$p
#[1] 0.2304269

#$f.gamma
#[1] 0.1002716

#$f.lambda
#[1] 0.03428872

#$f.epsilon
#[1] 0.00456466

#$f.eta
#[1] 2.109682

hist(log_returns_JT$transformed, freq = TRUE, label=TRUE)

curve(dnorm(x, mean=mean(log_returns),
               sd=sd(log_returns)), add=TRUE, col="darkblue", lwd=2)

curve(dnorm(x, mean=mean(log_returns_JT$transformed),
               sd=sd(log_returns_JT$transformed)), add=TRUE, col="red", lwd=2)

Fn = ecdf(log_returns_JT$transformed)
plot(Fn,
   main="CDF of log_returns_JT",
   xlab="log_returns_JT",
   ylab="Cumulative Frequency")
qqnorm(log_returns_JT$transformed)
qqline(log_returns_JT$transformed)

shapiro.test(log_returns_JT$transformed)
#Shapiro-Wilk normality test

#data:  log_returns_JT$transformed
#W = 0.99465, p-value = 0.5217

ad.test(log_returns_JT$transformed)
#Anderson-Darling normality test

#data:  log_returns_JT$transformed
#A = 0.48099, p-value = 0.2304

(adt.p<-RE.ADT(log_returns_JT$transformed)$p)
# [1] 0.2304269

Here are the binned histogram of frequencies, cumulative frequency counts, and qq-plot after the Johnson's transformation:

I have applied the normality hypothesis test (Anderson-Darling test) and the null hypothesis can't reject at level 0.05 (p-value is 0.2304269).
Edit. I'm reading the papers recommended by @user25459 but, unfortunately, I don't find the answer for my question yet.
Questions. Is it possible to use a normal copula to modeling dependence and set the marginal distribution as the normal distribution?
After the modeling I'm going to apply the inverse transformation.
Comment: From what you have told us we have no reason to believe in a particular form for the density. One thing about goodness of fit tests to keep in mind is that in small samples it is difficult to reject normality if the true distribution is nearly normal .  In very large samples it is easy to reject normality because even small departures from normal can be detected.  It may be that you only need to be close to normal to meet the requirement for statistical methods that assume normality will work well. In your case  250 is a moderate sample size.  The histogram looks approximately normal.
Comment: To add to Glen_b point rejecting normality doesn't tell you that the distribution is not normal either.  The fact is models are only approximations and if we say the "true" distribution has to be normal to apply normal theory we would be unable to apply much of parametric statistics.  The same can be said for goodness of fit tests for the exponential distribution or any other parametric family.  Results of goodness of fit tests deserves careful thought and is not the sole determinant of what inferential statistical tests should follow.
Comment: Also you only barely rejected at the 0.05 level and the Anderson-Darling statistic provide one of the most powerful tests when the null distributionis normal.
Comment: It's easier to see the departure from normality with a QQ plot rather than an arbitrarily binned histogram. Also, please indicate what the analytic goal is because then one can evaluate if the observed departure from normality is even relevant.
Comment: what is the data on  axis  A. Further, may be you should give graph  ndicating cumulative frequency.
Comment: @MichaelChernick, I have applied the Kolmogorov-Smirnov test and the null hypothesis can't rejected at level 0.05 (p-value is 0.21769). But the null hypothesis should be rejected based on Shapiro-Wilk test (p-value is 0.245e-4), Chen-Shapiro test (5% critical value is 0.00356).
Comment: @subhashc.davar, my data is log-returns of a finance time series.
Comment: Kolmogorov-Smirnov test is not designed to have high power against any particular family of distributions.  So it should not be surprising that it has a larger p-value than Shapiro-Wilk or Anderson-Darling which are designed to have high power for rejecting normality.  I am a little surprised that the p-value for the Shapiro-Wilk is so small. Don't think of statistical significance as black and white.  There is nothing magic about 0.05 to say that every p-value below 0.05 is significant and the all tests of the same hypothesis will agree.
Comment: Failure to reject does not tell you that the null is true (i.e. a goodness of fit test failing to reject normality does not say that the distribution is normal).
Comment: please a codensed glimpse of data alongwith purpose of study.
Comment: @subhashc.davar, I have specified the purpose of a study and uploaded the data.
Comment: @Nick For a quick and practical answer (leaving all theoretical discussion aside), try the generalised error distribution.
Comment: @mugen, could you please provide a link with an example of the generalized error distribution. I have found the function gedFit() from the fGarch package: require(fGarch)

## rged -
   set.seed(1953)
   r = rged(n = 1000)

## gedFit -
   gedFit(r)
Comment: @Nick See [rugarch](https://cran.r-project.org/web/packages/rugarch/). The vignette explains the basic theory and the documentation explains the use of the package. If you'd like to use the distributions outside a GARCH model, see [these functions](https://github.com/cran/rugarch/blob/master/R/rugarch-distributions.R).
Comment: @Nick I did a manual look at the data, and the hyperbolic secant distribution is a reasonable fit, see answer as to why in the answer below.  There is a bulge on the right which could be random noise, or could be the sign of multiple effects.  Please read the paper as to why the hyperbolic secant distribution would be a decent fit.
Comment: @DaveHarris, thanks you for comment, I have updated the question. Could you please clarify your comment?
Answer: There is now a proof that derives the complete set of asset and liability classes, though all are mixture distributions.   You can find it at https://ssrn.com/abstract=2828744
The reason for the mixture is bankruptcy risk, merger risk, liquidity costs, and a stochastic budget constraint. Some debt instruments can be modeled using the normal or lognormal, but no equity-type asset can have a first moment.  Some assets, such as antiques, have the ratio of two Gumbel distributions as its distribution so there is no analytic representation.
Generally, there is no distribution,  although for the broad population of US equities, its remarkably close to a truncated Cauchy distribution.  Multiply it by a very wide survival function and you'll get an extraordinarily good fit.   You can think of a trade as a survivor of the budget constraint because,  of course, not all orders fill.
The fundamental argument of the paper is that returns are not data, instead prices are data and returns are a  transformation of prices.  In particular,  it is a future value divided by a present value with an appraisal error upon entry and exit, minus one.  As such, all returns are ratio distrubutions.   Zero coupons differ in that the payoff is essentially fixed, so part of the ratio is a constant.
For completeness,  a population study was done on US trades from 1925-2013 to test the most important claims.  It can be found at https://ssrn.com/abstract=2653151
edit  I saw after my post it was log returns.  It will be a potentially ugly transformation of the hyperbolic secant distribution.   Really we should quit using the log approximation,  the only reason the process started was the need to use slide rules.
edit 2 I crudely mapped your bins in Excel and it was close to the hyperbolic secant, though probably skewed.  You can do the Johnson transformation because you can map any univariate standard distribution to any other univariate standard distribution, but you need to be careful that you do not forget that properties that govern are the properties of the untransformed data.
Consider the simplest possible case, the unrealistic case of Markowitz.  The distribution of returns in a universe with only two equal sized equity assets would be $$\frac{1}{2\pi}\frac{\gamma}{(\gamma^2+(r_1-\mu_1)^2+(r_2-\mu_2)^2)^{1.5}}.$$  You could map this distribution to any bivariate distribution, but look closely at its properties.
$\gamma$ is not a vector, it is a scalar.  If you add dimensions $\gamma$ will become $\gamma'$, but $\gamma'$ is still a scalar.  The likelihood has spherical uncertainty and the density has spherical errors.  You can map this to a normal distribution with a covariance matrix, but it will not be unique because you are mapping to a distribution with more parameters than exist in the raw data.
Let's imagine instead you map a Cauchy to a normal, with zero covariance on the off diagonals, that is $\sigma_{i,j}\equiv{0},\forall{i,j}$ and equal variances to assure circularity, that is $\sigma_{i,i}\equiv\sigma_{j,j},\forall{i,j}$.  In the normal distribution, asset 1 and asset 2 are independent, but in the raw data they are not independent.  You have preserved the shape, but not the dynamics.  The Cauchy distribution is an example of a distribution where none of the dimensions are correlated, but none are independent.
There is no theoretical reason to not map the raw data to a copula function, but the "how" to do that is something I cannot answer.  I have done very light reading on the topic as I have thought about it myself since the Cauchy distribution doesn't really appear except in a mixture, but I read enough to realize I would have to spend a lot of time on the topic.
My preference to solve the problem, if I ever get to it, would be to note that things such as bankruptcy and liquidity costs do covary with underlying factors and so I would use Bayesian model selection and tools such as marginalization to separate out the components.
Comment: I take issue with the general call to stop using logarithms. Logarithms have more uses than just ease of use with slide rules. Whenever you think in terms of multiplicative effects rather than additive effects the logarithm automatically pops up. The slide rule was invented because the logarithm is such a useful tool, not the other way around.
Comment: I agree with @Maarten Buis. The usefulness of a logarithmic transformation is completely independent of the technology you might use to implement it.
Comment: @MaartenBuis and Nick I have no issue with the use of the logarithm where it serves a theoretical role, as it does in economic growth models or in studying elasticity, but that isn't why it is generally used in economics.  If you asked most "why did you use the log?" the answer would likely be that it was done because the last person did it.  If economists were thinking in terms of multiplicative effects, that would be a different issue.
Comment: The logarithmic approximation is not harmless in economics, because $\exp(\beta_{log})\ne\beta$ you would hope the shift in values would be very small, but it is not.  For the population of returns in the CRSP universe, essentially every end of day trade from 1925-, the different between $\exp(\mu_{log})$ and $\mu$ is two percent per year and it understates risk by $\pm{4}\%$.  If you are going to do any transformation, there should be a substantive reason.
Answer:
Questions. Is it possible to use a normal copula to modeling dependence and set the marginal distribution as the normal distribution?
After the modeling I'm going to apply the inverse transformation.

As a basic principle, it's certainly possible to transform margins to approximate normality and then fit a multivariate Gaussian, which is in effect fitting a Gaussian copula to the dependence relationship.

I don't know how to specify a distribution to modeling dependence with a copula.

Perhaps I misunderstand your intent here, I'm not sure; the point of a copula is to fit the dependence structure independent of the marginals. Usually in copula work this is achieved by transforming the marginals to approximate uniformity (via one of several different approaches) then fitting some copula, but in the case of the Gaussian copula that's not necessarily the easiest approach (nor even the most common).
I see no particular issue with using Johnson transformations if they're adequate to achieve very near normality. It shouldn't be worse than some of the other approaches people use.

However, for asset returns usually the Gaussian copula is regarded as unsuitable because the tail dependence in the Gaussian copula goes to 0, while in real asset returns can show substantial tail dependence (in particular, if the market goes south, like say in the GFC, correlated assets will tend to see very extreme results at the same time -- the Gaussian copula can't capture that.
Sometimes the t-copula is used to model this, since it does have tail dependence, but it doesn't capture the asymmetry (the dependence is not equally strong in the upper and lower tail). It may be adequate for some purposes, however.
Comment: Thanks for answer, for instance, in the study: Davari-Ardakani H. et al (2016) Multistage portfolio optimization with stocks and options. International Transactions in Operational Research. 23 (3), 593–622. onlinelibrary.wiley.com/doi/10.1111/itor.12174/pdf the Johnson transformation was used and its is one of the main parts of a simulation.
Comment: @Nick as I suggest in my answer, it's not the Johnson that concerns me ("I see no particular issue"), it's the Gaussian copula when modelling dependence in returns that worries me.


Paypal: technique to obtain buyer's country for establishing postage prices without introducing extra steps

Question: This is probably a stupid question, I think I'm having a mental block.
I want to use Paypal's express checkout for buyers to make purchases with as few steps as possible (also using Paypal's optional  account feature). The problem is: for me to direct the user to paypal, I will need to have calculated shipping costs, which depends on the country they're in. Obviously I cannot know this unless I have been told one way or another. What are common solutions to this problem?
Ideas:

Use their IP address. Not reliable - various types of proxies, VPNs, anonymizers etc.
Have the user select their country from a drop-down box before I redirect them to paypal
Force them to log in using Paypal's Identity service before calculating postage
Use 1. or 2. and once payment is received, if country is different to expected,

Refund buyer the difference in postage cost or request an additional payment. (Hardly ideal)

Similar to 4., instead of "Sale Payment Action for Express Checkout" Use 1. or 2. in conjunction with one of Paypal's delayed express checkout payment methods and if necessary reduce or increase the amount charged:

Authorization Payment Action for Express Checkout
Order Payment Action for Express Checkout

I'd like to know what solutions other developers have chosen - maybe I've missed an idea. As a consumer, I cannot remember seeing solutions to this.
Answer: The general way people do this is simply to use GetExpressCheckoutDetails to obtain the buyer's shipping address, apply any shipping/tax as necessary, and display a final review page on your site that the user would confirm before calling DoExpressCheckoutPayment.
If you want to eliminate the additional review page (PayPal's and then your own) you can use the Instant Update API.
In this case you would include an additional parameter on the URL when you redirect to PayPal (useraction=commit) and this will change the button on the PayPal review page to say Pay instead of Continue.
Also in your SEC request you'll include the CALLBACK parameter and include a URL to your callback listener.  PayPal's review page will POST the buyer's shipping address to this callback URL so that you can receive the data, calculate shipping and tax accordingly, and send a response of those options back to the PayPal review page.  This will populate the PayPal review page's drop down menu for the shipping option and the user can choose accordingly.
This method allows you to utilize the PayPal review page entirely and finalize the payment there so that the only thing the user sees once they're returned to your site (after you call DECP) is a final thank you / receipt page.
Comment: Thoroughly informative and helpful answer, very much appreciated, thank you!


UnicodeDecodeError when parsing XML on mac but works on PC

Question: When parsing a XML file with:
from lxml import etree

with open('cortex_full.xml', 'r') as infile:
    root = etree.parse(infile)

I am getting the UnicodeDecodeError below.  This only happens on my Mac though - if I parse the same file with the same script on my work PC, everything works fine.
File "/Users/Desktop/CPET/xml_test2.py", line 5, in <module>
    root = etree.parse(infile)
  File "src/lxml/lxml.etree.pyx", line 3442, in lxml.etree.parse (src/lxml/lxml.etree.c:81701)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118888)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:119171)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117959)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:112686)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105881)
  File "src/lxml/parser.pxi", line 702, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:107548)
  File "src/lxml/lxml.etree.pyx", line 324, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:12152)
  File "src/lxml/parser.pxi", line 373, in lxml.etree._FileReaderContext.copyToBuffer (src/lxml/lxml.etree.c:103210)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 783: ordinal not in range(128)

This seems to be quite a common occurrence given the number of threads on here, however none of the suggested fixes seem to be working for this instance. Any ideas for getting it to work? Full XML file here
Comment: `with open('cortex_full.xml', 'r', encoding='utf-8') as infile:`
Comment: are you using the same version of Python on both systems?
Comment: Yes, most up-to-date version of Python3 and lxml on both.
Comment: Interestingly, I tried that yesterday and it didn't work but when I tried it just now it works. I must have typed something different the first time.
Answer: Posting an answer that worked for me for future reference.
Credit goes to @Burhan Khalid for the answer.
Need to set encoding to utf-8 when opening the xml file.
with open('cortex_full.xml', 'r', encoding='utf-8') as infile:


Let X and Y be independent and identically distributed random variables with probability density function. Then P(y<x<2)

Question: Let X and Y be independent and identically distributed random variables with probability density function
f(x)= \begin{cases}
       e^{-x}  & x>0\\
        0 & otherwise\\
    \end{cases}
Then P(y<x<2)  =
My attempt at the solution:\begin{align}
p(y<x<2)=p(x<2)p(y<2)-p(y \geq x)
=\int_{0}^{2}e^{-x}dx\int_{0}^{2}e^{-y}dy -\int_{0}^{2}\int_{0}^{y}e^{-x}dx dy
\end{align}
But doing this is not giving me the correct answer. Can somebody please tell me where I am going wrong and give me the correct approach
Comment: It looks like you're missing an $e^{-y}$ in the last integral.
Comment: Also, notice that the last term is $P(X\le Y\le 2)$, not $P(X\le Y)$. And $P(X=Y)=0$, so there's a helpful symmetry to the problem since X and Y are independent and identically distributed.
Answer: With these types of problems it is always helpful to draw pictures. First, we have for the joint density of $X$ and $Y$:
$$
f_{XY}(x,y)=e^{-x}e^{-y},\quad 0<x,y<\infty.
$$
Now draw the region $0<y<x<2$:

The probability in question can now easily be written as an integral in the form
$$
\mathsf P(Y<X<2)=\int_0^2\int_0^x e^{-x}e^{-y}\,\mathrm dy\mathrm dx=\frac{\left(e^2-1\right)^2}{2 e^4}\approx 0.75.
$$


Is normalizing/standardizing features and target separately a good method?

Question: Suppose I scaled the features and target by creating separate objects, like this
<code>X_scaler=MinMaxScaler(feature_range=(0, 1))
Y_scaler=MinMaxScaler(feature_range=(0, 1))

X_sca = X_scaler.fit_transform(X)
Y_sca = Y_scaler.fit_transform(Y)
</code>
I am working with sequence prediction with three inputs and two outputs. Is there any problem with this approach? If yes, please give the preferred method.
Answer: To answer your questions

Is there any problem with this approach?

None at all.

Is normalizing/standardizing features and target separately a good method?

Depends on what you want to do. If you want to use the scaler in a pipeline or something like that, where you need two scalers, its a good practice.
If you don't have any special reason for wanting to scale them separately, it's more efficient to scale them with the same scaler.
In most applications you don't need to scale the target at all.


Ruby on Rails 5.0 Table removal

Question: I am currently working on a project and I have messed up my code. I went back and deleted the git branch. I am now going back through and adding all of the code back.
I need to create a db migration but the migration that I need to create says it already exists. I don't see how this could be possible.
My question is how would I get rails to just overwrite the table?
Answer: Although you deleted your git branch and local code, if you didn't delete your database, it remains the same so your table still exists.
You can create a migration to drop a table.
Code from this answer.
<code>rails generate migration DropProductsTable

class DropProductsTable < ActiveRecord::Migration
  def up
    drop_table :products
  end

  def down
    raise ActiveRecord::IrreversibleMigration
  end
end
</code>
I don't know what database you are using, but if the table is the same as what you want to add you may not even need to drop it.
Comment: I see. I am currently working through a bootcamp course and I messed up the branch so I deleted it. The database I am using is SQLite3. So if the data for the table is the same as the first one I can just leave it?
Comment: This was my out put

redhoodjt:~/workspace/saasapp (contact_form) $ rails db:migrate
== 20170223181245 CreateContacts: migrating ===================================
-- create_table(:contacts)
rails aborted!
StandardError: An error has occurred, this and all later migrations canceled:
Comment: Yes. Your database hasn't been touched even though you deleted the branch so the table is still there. You could also use ` rake db:rollback` to undo the last migration you made, or `rake db:rollback STEP=n` (n being the number of migrations to rollback) depending on when you created this table. Then `rake db:migrate` with your new migrations that you've created.
Comment: The only problem with that is that for some reason the error that I am being given says that all later migrations canceled.
Comment: Yes. If you have 3 migration files and the first one gives you the error because the table is already there, the 2 that come after it will get cancelled. You need to resolve the issue with the problem migration by dropping the table/rolling back or removing the migration file. Is the original migration that you made for this table still there, or did you delete it?
Comment: I deleted it. At least I think that I did.
Comment: Ok I deleted it and that fixed the issue. Thank you.
Comment: But now I have another issue. When I try to see if my submission form works I get an error saying:

No route matches [POST]"/contacts/new"

When I go to my routes.rb file though I have resources :contacts in there.
Comment: Without seeing your code it is impossible to give you an answer for that.
Comment: Ok. I can post it. How do I post code so it shows up like code? And from what file or files do you need the code from?
Comment: Lets continue this discussion in [chat](http://chat.stackoverflow.com/rooms/136554/no-route-matches-post)
Comment: My reputation isn't high enough to chat.
Comment: Ok that's a pain. You should post this as a new question then, add the form code from your view, routes, and new and create controller methods

Ruby on Rails 5.0 Table removal


How to point one domain to deep page of another domain with apache or htaccess

Question: I have two domains (a.com and b.com) and a.com is running on my apache server.  I want web browsers that try to visit b.com to end up at a.com/?b (transparently, no redirect, and no second virtual host if possible).
I see that within the virtual host config i can rewrite (as shown here) but it's not clear and there are no examples.
Can someone propose the simplest way to solve this, ideally without another virtual host?

UPDATE:
I'm trying to solve it something like this.  I think I'm close but a bit lost:
<virtualhost XXX.XXX.XXX.XXX:80>
        ServerName   a.com
        ServerAlias  b.com
        RewriteEngine On
        RewriteCond %{HTTP_HOST} (.*)\b\.com
        RewriteRule ^(.*) http://%1a.com/?b [L]
</VirtualHost>

Comment: Somebody has to set up a virtual host for this, whether you or someone else.
Comment: Is it possible to share the same virtual host as a.com ?  And if so, how would I direct to the right page?
Comment: Yes, but it makes your setup much more complicated. Just make a new virtual host. It's simple, easy and fast.
Comment: Would I need to redirect (eg 301) if I use a new virtual host? I want this to be transparent
Answer: You could use Reverse HTTP proxy configuration on your apache server.
Check this out
Your http server will be essentially serving content of another server.
e.g.:
<VirtualHost *:*>
    ProxyPreserveHost On
    ProxyPass / http://a.com:8080/
    ProxyPassReverse / http://a.com:8080/
    ServerName b.com
</VirtualHost>

Comment: And it will need a virtualhost, just what the OP doesn't want.
Comment: There is nothing bad with virtualhost, in this case it will not redirect the client, your webserver will essentially be able to serve clients requests in the following fashion. Cllient --> Your server A (Server A connects to server B for content) --> ServerB, Server B responds to server A, then the content is provided to the client by Server A. So in the headers clients will see only ServerA's IP.


URL Rewrite trouble

Question: I have a website develop on Drupal 7 on IIS + MSSQL. I cannot update or create content when clean urls is active. The website return to edit form when i submit any change or new content.
If the cleans url is disable then the content is saved.
Please help me with this trouble.
Regards.
Answer: Place the following debug statement in your index.php to debug your request:
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
debug($_GET['q']); //debug querystring
menu_execute_active_handler();

Now save/edit a node with and without friendly-urls.
If the values differ from each other, your rewrite-rules do not match.
If the values are the same, something else causes the problem.
If so, its hart to say, what causes the problem. One hint: page requests are GET-requests, save/edit requests are POST-requests
Comment: this sounds like a really helpful tip, upboat for you!
Comment: Hi, i try with the debug statement from larrydahooster, the debug return the node number for content pages, for admin pages return the clean url of admin page (eg. admin/config/system/site-information) but dont save / update the data and return to edit mode in the same url. If i turn off the clean urls then edit content is successfull.

firebug is returning this error when the page load :
Comment: `code` "TypeError: c.defaultView.getComputedStyle(...) is null
[Break On This Error]

...d*)(?:n([+\-]?\d*))?/.exec(a[2]==="even"&&"2n"||a[2]==="odd"&&"2n+1"||!/\D/.test..." `code`

Really the trouble is with post whith clean url on.
Comment: So basically your Rewrite Engine Works? Your error looks like a JavaScript-error. Really strange behavior... Are you working with a Clean-Install of Drupal?
Comment: hi again, for aisle the trouble i disable the javascript on the web browser and the trouble persist. The issue is only with the POST request, because the display of pages is ok (GET requests). We have a drupal 7.2 instalation, with omega theme.
Answer: Google is your friend. From IIS.NET:

By default, Drupal uses query string parameters for all the URLs it
  generates for your Web site. With the URL Rewrite Module, however, you
  can create "clean URLs."
IIS 6.0 does not have URL rewriting capabilities, so you can use a
  third-party URL rewriting product such as ISAPI Rewrite or Ionics
  ISAPI Rewrite Filter.
IIS 7 and above have URL rewriting support, which can be enabled by
  installing the URL Rewrite Module.
The following instructions describe how the URL Rewrite Module for IIS
  7 and above can be used to enable clean URLs in Drupal.

Open the Web.config file located in C:\inetpub\wwwroot\drupal folder. If you do not have a Web.config file in the WordPress
  directory, create it.
Copy and paste the following XML code into this file.

<?xml version="1.0" encoding="UTF-8"?>
    <configuration>
        <system.webServer>
            <rewrite>
                <rules>
                    <rule name="Drupal Clean URLs" stopProcessing="true">
                        <match url="^(.*)$" />
                        <conditions>
                            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
                            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
                        </conditions>
                        <action type="Rewrite" url="index.php?q={R:1}" appendQueryString="true" />
                    </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Save the web.config file.

Then you can properly configure CleanURLs in drupal at /admin/settings/clean-urls
Comment: Hi tenken, this rules already into the web.config of my drupal website, in fact the website show pages with clean urls, the trouble is that when clean urls are active we cannot create, update content or modify settings, the website return to edit mode and don't show errors.

URL Rewrite trouble


JavaScript -- insert "\r\n" at every line break in string

Question: I'm trying to use this CSV parsing library to parse my CSV, but it seems that this library needs "\r\n\" to be at the end of every row.
Right now, my CSV looks like this:
Bob, Smith, Cats,
Annie, Johnson, Dogs

How do I add the string "\r\n\" right after "Cats" and "Dogs"? Or is this unnecessary? Thanks!
Comment: Are you sure that this is a valid csv?
Comment: Once I read it in using the native FileReader API, the output is that string.
Comment: If that is your csv, it already has an `\r\n`, or, at lest an `\n`.  That is what puts the next portion of the string on the next line.  And, i doubt it is supposed to have quotation marks.
Comment: If you do a `console.log(yourCsvData);` ..what does it print?
Comment: The `\r\n` aren't expected to appear within `.csv` files. They are only used in the demos because they're defined with string literals in code rather than with separate files. [MDN: Using special characters in strings](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Values,_variables,_and_literals#Using_special_characters_in_strings). The sequence in a string represents the [default Windows newline](https://en.wikipedia.org/wiki/CRLF#Representations), CRLF.
Comment: If you created the file from within Windows, chances are that your file already contains "\r\n" at the end of line 1. If that's the case, your parser [should] see 2 lines. I noticed that you removed your quotes, but when you added the comma to the end of the first line, you told the parser that there is a blank field in row one, for a total of 4 fields, whereas line 2 only has 3 fields. I must add that it is simpler to write your own CSV parser than to learn somebody else's, especially if your data is already normalized.
Answer: Check what works for your environment!
You could change the default '\r\n' to '\n' like below:
var csv = new CSV(data, { line: '\n' });

Hope this helps!
Comment: This makes sense! In the example above, what line break should I feed in? Will the parser interpret "\n" as a line break or a string?
Comment: Thanks @Robert! I have not worked with the parser myself, but from the looks of documentation on github, it should interpret as line breaks.
Answer: You may try something like this:
var text = "Box, Smith, Cats\nAnnie, Johnson, Dogs";
text = text.split( "\n" ).join( "\r\n" );

JsFiddle
Answer: In this case, the documentation indeed says you can simply specify the line characters yourself with the line option as already pointed out in the other answer.
Just providing some additional info on this:
\r stands for carriage return
\n stands for line feed (newline)
Depending on which program you used to generate or manually write the CSV file, when you hit Enter this will produce either \n or \r\n
You can 'pre-cure' this with a text editor like Notepad++, Sublime Text, Brackets, etc. which have a regex replace feature. (Ctrl + H to bring it up in most cases). Most notably, Notepad++ allows you to see all characters in the View menu (\r will show up as a block with CR & \n will show up as a block with LN).
Simply check the Regex checkbox or icon and replace all \n with \r\n or vice versa.


Unable to start activity ComponentInfo{}: java.lang.NullPointerException

Question: Estou com dificuldade de encontrar a solução para esse problema, no código quero passar da activity "MainActivity" para "PlantaList", mas na hora de testar o aplicativo na máquina virtual o app fecha depois dessa ação e no terminal exibe o seguinte erro:

-17 16:05:44.130 2082-2082/com.example.adriano.campanha_flora_pre_alpha E/AndroidRuntime: FATAL EXCEPTION: main
         Process: com.example.adriano.campanha_flora_pre_alpha, PID: 2082
          java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.adriano.campanha_flora_pre_alpha/com.example.adriano.campanha_flora_pre_alpha.PlantaList}: java.lang.NullPointerException
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2195)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
        at android.app.ActivityThread.access$800(ActivityThread.java:135)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:136)
        at android.app.ActivityThread.main(ActivityThread.java:5017)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:515)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
        at dalvik.system.NativeStart.main(Native Method)
         Caused by: java.lang.NullPointerException
        at com.example.adriano.campanha_flora_pre_alpha.PlantaList.onCreate(PlantaList.java:32)at android.app.Activity.performCreate(Activity.java:5231)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2159)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
        at android.app.ActivityThread.access$800(ActivityThread.java:135)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:136)
        at android.app.ActivityThread.main(ActivityThread.java:5017)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:515)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
        at dalvik.system.NativeStart.main(Native Method)

Segue Android Manifest:
<?xml version="1.0" encoding="utf-8"?>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

<application
android:allowBackup="true"
android:icon="@drawable/campanha_flora_icon"
android:label="@string/app_name"
android:supportsRtl="true"
android:theme="@style/AppTheme">
<activity android:name="MainActivity">
    <intent-filter>
        <action android:name="android.intent.action.MAIN" />

        <category android:name="android.intent.category.LAUNCHER" />
    </intent-filter>
</activity>
<activity android:name=".PlantaList"/>
<activity android:name=".ConsultaAdapter" />
<activity android:name=".TelaCadastro"/>

</application>

Segue Main Activity
package com.example.adriano.campanha_flora_pre_alpha;

import android.content.Intent;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.view.View;
import android.widget.Button;

public class MainActivity extends AppCompatActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);

Button botaocadastrar = (Button) findViewById(R.id.btnCadastrar);

botaocadastrar.setOnClickListener(new View.OnClickListener() {
    public void onClick(View v) {
        startActivity(new Intent(MainActivity.this, TelaCadastro.class));

    }
});

Button botaoconsultar = (Button) findViewById(R.id.btnConsultar);

botaoconsultar.setOnClickListener(new View.OnClickListener() {
    public void onClick(View v) {
        startActivity(new Intent(MainActivity.this, PlantaList.class));

    }
});

Button botaopreferencias = (Button) findViewById(R.id.btnPreferencias);

botaopreferencias.setOnClickListener(new View.OnClickListener() {
    public void onClick(View v) {
        startActivity(new Intent(MainActivity.this, TelaCadastro.class));

    }
});
}
}

Segue PlantaList:
package com.example.adriano.campanha_flora_pre_alpha;

import android.database.Cursor;
import android.os.Bundle;
import android.support.annotation.Nullable;
import android.support.v7.app.AppCompatActivity;
import android.widget.GridView;

import java.util.ArrayList;

public class PlantaList extends AppCompatActivity {

GridView gridView;
ArrayList<Planta> list;
ConsultaAdapter adapter = null;

@Override
protected void onCreate(@Nullable Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.tela_consulta);

gridView = (GridView) findViewById(R.id.gridView);
list = new ArrayList<>();
adapter = new ConsultaAdapter(this, R.layout.tela_consulta_2, list);
gridView.setAdapter(adapter);

// get all data from sqlite
Cursor cursor = TelaCadastro.sqLiteHelper.getData("SELECT * FROM PLANTA");
list.clear();
while (cursor.moveToNext()){
    int id = cursor.getInt(0);
    String NomePopular = cursor.getString(1);
    String Especie = cursor.getString(2);
    String Familia = cursor.getString(3);
    String TipoDeRaiz = cursor.getString(4);
    String CorDeFlor = cursor.getString(5);
    byte[] image = cursor.getBlob(6);

    list.add(new Planta(NomePopular, Especie, Familia, TipoDeRaiz, CorDeFlor, image, id));
}
adapter.notifyDataSetChanged();
}
}

O que pode ser?
Comment: Verifique se `gridView`ou `cursor` é nulo.
Answer: Pelo trecho de código
Cursor cursor = TelaCadastro.sqLiteHelper.getData("SELECT * FROM PLANTA");

que está na PlantaList, imagino eh que o sqLiteHelper não esteja inicializado.


Variavel via POST para sed

Question: Eu estou com um raspbian rodando num raspberry pi3, estou fazendo um sistema em cima dele que necessita que a pessoa escolha o hostname digitando ele num campo. E para trocar o hostname do rasp eu preciso alterar dois arquivos dentro da pasta etc. Já consegui fazer essas alteração pelo comando sed, porém como quero que a pessoa escolha o hostname, fiz um formulário por PHP e queria pegar via POST essa variável e repassar para o comando sed. Segue o códigos.
<link rel="stylesheet" type="text/css" href="css/style.css">

<form method="POST">

<input  type="text" name="hostname">
<input type="submit" name="Comentar" value=" Salvar">

</form>
<?php

if(isset($_POST['Comentar'])){
        echo "botão foi clicado"."
";
        $SEU_HOSTNAME = $_POST["hostname"];
        echo  "hostname digitado: ".$SEU_HOSTNAME;

    shell_exec ('

        cd /etc/

        sudo sed -i "s/nome/${SEU_HOSTNAME}/" hosts
        sudo sed -i "s/nome/${SEU_HOSTNAME}/" hostname

    ');

    }

?>

Fazendo desta forma, aparece somente as { } no arquivo que modifico.
Alguém saberia o que pode ser isso?
Comment: Osmar, [não poste código como imagem](https://pt.meta.stackoverflow.com/a/5485/132), o site tem suporte para os códigos, basta utilizá-los. Você pode fazer o [tour] para aprender o básico do site, ler o guia de [ask] e acessas a [help].
Answer: Olá,
Para que o arquivo PHP consiga alterar o arquivo, ele deve ser executado como sudo e tente mudar o código para o seguinte:
$command1 = 'sudo sed -i "s/nome/'.$SEU_HOSTNAME.'/" hosts';
$command2 = 'sudo sed -i "s/nome/'.$SEU_HOSTNAME.'/" hostname';

shell_exec("cd /etc/");
shell_exec($command1);
shell_exec($command2);

Espero ter ajudado, abraço!
Comment: Exatamente isso, funcionou perfeitamente.
Answer: Fazendo desta maneira também consegui resultado. Depois de testar o código acima, pesquisei mais um pouco e achei esta solução.
$HOSTNAME=escapeshellarg("s/nome/$SEU_HOSTNAME/");
shell_exec ("
    cd /etc/
    sudo sed -i $HOSTNAME hosts
    sudo sed -i $HOSTNAME hostname
");

Variavel via POST para sed


ios nsdictionary low avg high

Question: Right now I'm taking my NSDictionary and running a loop of all values to find the low, high, and calculate average.
Since I'm new to IOS I wanted to ask if there is a better way to do this. Is there? Thanks.
Comment: Edit your question and paste in your code.
Answer: The exact answer to this question depends on what type of objects are in your NSDictionary. The answer in general is that you can probably use Key-Value Coding operators to do what you want. Here is an example:
NSNumber *one = [NSNumber numberWithInt:1];
NSNumber *two = [NSNumber numberWithInt:2];
NSNumber *three = [NSNumber numberWithInt:3];

NSDictionary *dictionary = [NSDictionary dictionaryWithObjectsAndKeys:one, @"apple", two, @"orange", three, @"pear", nil];
NSLog(@"The max number is %@", [[dictionary allValues] valueForKeyPath:@"@max.intValue"]);
NSLog(@"The min number is %@", [[dictionary allValues] valueForKeyPath:@"@min.intValue"]);
NSLog(@"The mean is %@", [[dictionary allValues] valueForKeyPath:@"@avg.intValue"]);

This is probably no more efficient than enumerating things yourself, but it is more concise.
If the contents of your NSDictionary are objects of a custom class that have an NSNumber as a property, you can still use this method by placing the property name after the @avg, e.g. @avg.myProperty. Note that these valueForKeyPath methods return NSNumbers.
Answer: There isn't a better way to do it than how you're already doing it by enumerating the dictionary. This is mainly because NSDictionary contains opaque objects. They might be NSStrings, they might be NSNumbers or they might be CHZLolCats. They can be anything. So it wouldn't make sense to have a low, high and average method as not everything has the concept of a low, high and average.
However if you showed us some code we might be able to offer specific advice on the actual method of enumeration.

ios nsdictionary low avg high


Find all solutions to $x^3-x^2+2x-2=0 \pmod {11}$

Question: I get that $11$ is a small number and that I could maybe do this by inspection, but I was wondering if there was a more intelligent approach?
Answer: Hint: you could try to factor
$$\begin{align}
x^2(x-1) + 2(x-1) &= 0 \pmod{11}\\
(x^2 +2)(x-1) &= 0 \pmod{11}
\end{align}
$$
Comment: Just hover your mouse over the "answered just now". You literally beat me by $1$ second. Cheers
Comment: @Sabyasachi: Wow, I am usually so slow. You answer is longer though.
Comment: Yes it is. It is basically the same, you should add $\pmod{11}$ though. (Yes I know, the modulo is present in the question, but someone in a hurry might misinterpret)
Comment: @Sabyasachi: Ok, I will do that...
Comment: Yeah it's all good now. Upvoted.
Answer: $$x^3-x^2+2x-2\equiv0 \pmod {11}$$
$$(x^2+2)(x-1)\equiv0\pmod{11}$$
Since $11$ is prime,
$$x-1\equiv 0\pmod{11}\implies x\equiv 1 \pmod{11}$$
OR
$$x^2\equiv-2\pmod{11}$$
Can you finish the last conrguence by yourself?
Comment: For the last congruence: 3 and -3. Why do you say "since 11 is prime", what if it wasn't?
Comment: @TMatija suppose $ab$ is divisible by $10$. Does that mean either $a$ or $b$ is divisible by $10$? Maybe, one is divisible by 2 and the other by 5?


Can I use ansible to startup jboss service in domain mode?

Question: I used sh /usr/local/jboss/bin/domain.sh manually, and jboss works fine.
When I use ansible to start jboss automatically, jboss log always shows "Container is down".
File /root/startup_jboss.sh :
#!/bin/bash
/bin/sh /usr/local/jboss/bin/domain.sh &

My startup.yml:
---
- name: Tranfser and Execute a script
  hosts: AuditReport

  tasks:
     - name: Tranfser the scipt
       command: chdir=/root /bin/sh startup_jboss.sh

I don't know why?
Could anyone help me?
Thanks.
Regards,
Cloud
Comment: Why aren't you having `ansible` run the same script that works for you manually?
Comment: Thanks your reply. I found can't use command module(can't run in the backgrond), I tried service module and it works fine.
Answer: The error you encounter is related to the fact that the context of the execution of the domain.shscript is different when you're logged in and when it is executed remotely by Ansible. See this SO question for more details.
You need to complete the /root/startup_jboss.shscript with the missing environment variables in order to work correctly. You can test it in the same conditions than ansible with a remote SSH command:
ssh root@<your node> /root/startup_jboss.sh
You may use Ansible to execute a shell script on a remote server but I don't think it's the right solution in your use-case.
Ansible is a Configuration Management tool that is designed to describe desired states for your resources (files, process, rights, ...) and ensure idempotency (only the changes that are required are done, you can run your script multiple times and you always get the same state). In your use-case, you'd like to ensure JBoss is running in domain mode on your remote server. The way I'd tackle it is to encapsulate the domain.sh script inside a init script to have it running as a service and eventually configure this service to run at server startup.
I don't know what is your linux distro but you may have a look here for an example of an init.d script for Debian.
In your startup.yml playbook you'd first copy the init.d script and then ensure your service is running (Ansible provides a native module to manage services).
Comment: Thanks for your reply. I trid to use service, and it works fine. Thanks!!
Comment: Nice! Can I ask you to mark your question as answered by my answer? (see [here](http://meta.stackexchange.com/questions/147531/how-mark-my-question-as-answered-on-stackoverflow) for how-to). THKS.


How can I limit the geofire quarry's range with security rules?

Question: So I am building an app witch returns the users in nearby.
This functionality is provided by the GeoFlutterFire library.
The problem is I don't want to let some guy to just dump all of the locations from the database with a single quarry.
So how can I limit the max range of the quarry with the firestore security rules?
Here is my user document structure
Answer: You won't be able to do this with security rules.  The rules you use to allow access to a collection for any GeoFire query will have to allow read access to all documents.  Once you allow that, you are allowing anyone to read all documents in the entire collection.
If you need to control the specific parameters to a query, you will need to force the user through a backend that will check the arguments, place limits on the query, and only return as many documents as they are allowed.  That said, someone could still call that backend with lots of different paramters to try to get all the documents, so all you've really done is make it more difficult.


One-Second Timer

Question: I need a timer that does X every second.
I made this, however it doesn't print anything until the program is terminated, I find that weird.
It prints everything after three seconds if you put three as the counter, and 100 if you chose that.
How do make it print every second and not all at once at termination?
int main()
{
    using namespace std;
    //Number to count down from
    int counter = 10;
    //When changed, a second has passed
    int second = (unsigned)time(NULL);
    //If not equal to each other, counter is printed
    int second_timer = second;
    while (counter > 0) {
        second = (unsigned)time(NULL);
        while (second != second_timer) {
            //Do something
            cout << counter-- << ", ";
            //New value is assigned to match the current second
            second_timer = second;
        }
    }
    cout << "0" << endl;
    return 0;
}

Comment: Using up all of the CPU while you wait for a timeout really isn't a solution. Anyway, this has nothing to do with timers. `std::cout` is buffered.
Answer: Add << flush where you want  to flush. I.e. change your printout to:
cout << counter-- << ", " << flush;
Answer: endl causes the buffer to 'flush' and be written out to stdout. You can add << endl; to your  cout << counter--, manually flush the cout stream using cout.flush();, or append << flush; to the end of your cout expression (thanks @Rob!)
For more info, the answer to  this question seems to go into more detail.

One-Second Timer


Erro na gravação de datas

Question: Seguindo na linha de Validar diferentes formatos de data
A função abaixo supriu minhas necessidades:
function validateDate( $date ){
   $europeu = date('Y-m-d', strtotime(current(explode(' ', trim($date)))));
   $brazil = date('Y-m-d', strtotime(str_replace('/', '-', current(explode(' ', trim($date))))));
   return ( $europeu != '1970-01-01' ? $europeu : ( $brazil != '1970-01-01' ? $brazil : null));
}

Saida:
validateDate('11/12/2014'); // MÊS/DIA/ANO -> 2014-11-12 OK
validateDate('31/10/2014'); // DIA/MÊS/ANO -> 2014-10-31 OK

Agora me deparo com mais um problema, ao gravar no DB utilizando essa função algumas datas (não todas) ficam assim:
0000-00-00
4537-11-13
7963-11-13
2014-10-29

A estrutura está assim:
Nome       Tipo Colação Atributos   Nulo    Padrão                    Extra
base_dataos  date                  Não  Nenhum wrap (padrão: none)

Inserindo manualmente pelo phpmyadmin a data grava corretamente, segue query gerada
INSERT INTO `banco`.`tabela` (`base_id`, `base_posto`,   `base_numeroos`, `base_numerosr`, `base_modelo`, `base_garantia`, `base_codays`, `base_codphi`, `base_dataos`, `base_full`, `base_upload`, `base_user`) VALUES (NULL, 'EMPRESA TESTE', '99999999999', '88888888888', 'NTRX', 'Na garantia', '33333', 'ap1212', '2014-11-13', 'N', CURRENT_TIMESTAMP, 'Admin');

A query usada no PHP
// dados pego de um arquivo excel
$data_os = validateDate( $data[5] );

$mysqli->query("INSERT INTO `tablea` (`base_posto`, `base_numeroos`, `base_numerosr`, `base_modelo`, `base_garantia`, `base_codays`, `base_codphi`, `base_dataos`, `base_user`)
    VALUES ('".$posto."','".$numero_os."','".$numero_sr."','".$modelo."','".$garantia."','".$codigo_posto_ays."','".$codigo_posto_phi."','".$data_os."', '".$userLogin."')")
    or die ( $mysqli->error );

Comment: Já vi pela edição da pergunta que estás a usar plicas para guardar a data na base de dados. Se fizeres `var_dump($data_os);` antes do objeto `$mysqli` qual o resultado que obténs?
Comment: Já tentou inserir uma data manualmente?
Comment: Sim @KhaosDoctor, na opção **inserir** do phpmyadmin add uma nova linha e gravou certo.
Comment: Ele mostra a query SQL?
Comment: O formato [`date`](http://dev.mysql.com/doc/refman/4.1/en/datetime.html) recebe os valores como `YYYY-MM-DD` na forma de uma _string_. Estás a inserir tipo: `'2014-10-29'`, ou seja, com plicas ?
Comment: @Zuul não entendi, o que á de errado? deveria ser tipo `VARCHAR` a coluna **base_dataos**?
Comment: @KhaosDoctor, complementei minha pergunta com sua dúvida
Comment: @Zuul, fui checar se tava importanto os dados corretamente conforme você sugeriu o `var_dump` e achei o BO. estou utilizando a class [PHPExcel](http://phpexcel.codeplex.com/) para importar dados de uma planilha e gravar no DB. vi que as datas no formato `11/12/2014 // MÊS/DIA/ANO` a célula carrega com a formato **Geral** retornando algo do tipo `41770.67140046296` dai as datas estão gravando errado. A questão agora é como importar a planinha no formato TEXT sem formato
Comment: @Hugo Você vai precisar achar as datas para poder converter em dias de novo. Talvez pegar os valores reais das células?
Comment: @KhaosDoctor, então.. abri um [POST](http://pt.stackoverflow.com/questions/40344/importar-planilha-com-formato-texto-utilizando-phpexcel) com essa dúvida
Answer: O problema era na importação da planilha, usando PHPExcel as datas no formato MM/DD/YYYY retornava um valor assim 41770.67140046296 a solução está na pergunta: Importar planilha com formato texto utilizando PHPExcel
Answer: Passei por um problema muito parecido com o seu. Pesquisando um pouco eu encontrei a seguinte solução, simples, que resolveu meu problema.
Eu precisava capturar uma data preenchida em um formulário no formato DD/MM/YYYY e transformá-la para o formato YYYY-MM-DD e inserir no banco. Ao inserir no banco, algumas davam problema. Então, para solucionar, eu capturei a data do formulário e corrigi com uma única linha:
$data = $_POST['data'];
$data_corrigida = implode("-",array_reverse(explode("/",$data)));

Comment: Obrigado por seu post @Abbominavell

Erro na gravação de datas


Calculate a multiple Ito integral

Question: Consider the following multiple Ito integral,
$$I(n)=\int\limits_0^t \int\limits_0^{t_n} \int\limits_0^{t_{n-1}} \dots \int\limits_0^{t_2}dW_{t_1} \, dW_{t_2} \, \dots \, dW_{t_n},$$
where $W_t$ is a standard Brownian motion.
$I(1)=W_t$
$I(2)=\frac{1}{2}W_t^2-\frac{1}{2}t$
$I(3)=\frac{1}{6}W_t^3-\frac{1}{2}tW_t$
The first 3 terms seem to be elegant.
I wonder whether there is a general term formula.
Answer: Such iterated integrals are closely related to Hermite polynomials.
If we define
$$H_n(t,x) = \frac{(-t)^n}{n!} \exp \left( \frac{x^2}{2t} \right) \frac{d^n}{dx^n} \exp \left(- \frac{x^2}{2t} \right),$$
then a straight-forward (but messy) computation shows that
$$dH_{n+1}(t,W_t) = H_n(t,W_t) \, dW_t \tag{1}$$
see this answer for details. You can easily compute the first Hermite polynomials $H_n$: $$H_1(t,x)=x \qquad H_2(t,x) = \frac{x^2}{2}- \frac{t}{2} \qquad \ldots.$$

Claim: $$I(n) = H_n(t,W_t). \tag{2}$$

We prove the assertion by induction. For $n=1$ it is obvious that $I(1)=W_t = H_1(t,W_t)$. Now suppose that $(2)$ holds for $n=k$, then we obtain from $(1)$ that $$H_{k+1}(t,W_t) = \int_0^t H_k(s,W_s) \, dW_s \stackrel{\text{ind. hypothesis}}{=} \int_0^t \left( \int_0^s \dots \int_0^{s_2} dW_{s_1} \ldots \, dW_{s_k} \right) \, W_s,$$ i.e. $(2)$ holds for $n=k+1$.

Calculate a multiple Ito integral


What is the correct step to execute execve()?

Question: It is required to use pipe(), fork(), execve() and dup() to implement a simple execution of terminal command with pipe for our homework. So I read about how dup and pipe manipulate the file descriptor, and I produced the code below.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void)
{
    int pfds[2];
    pipe(pfds);

    if (!fork()) {
        close(1);       /* close normal stdout */
        dup(pfds[1]);   /* make stdout same as pfds[1] */
        close(pfds[0]); /* we don't need this */
        char *k = {"echo", "one", "two", "three", NULL};
        execve("/bin/echo", k, NULL);
    } else {
        close(0);       /* close normal stdin */
        dup(pfds[0]);   /* make stdin same as pfds[0] */
        close(pfds[1]); /* we don't need this */
        char *k = {"wc", "-w", NULL};
        execve("/usr/bin/wc", k, NULL);
    }

    return 0;
}

It looks like there's nothing coming out by running the code, I am not sure what else do I need to make it work.
I am expecting out put of 3, as you will see by entering
echo one two three | wc -w in the terminal. I am using a MacOS by the way.
Comment: Note that you are passing the executed programs an empty environment.  That's not what you're supposed to do.  Either pass in a custom environment, or use the process's own environment (`extern char **environ;` outside any function — this is not declared in any POSIX header — and then use `environ` as the third argument to `execve()`, or  use `execv()` instead of `execve()` since you aren't tinkering with the environment.
Comment: @Angevil the dup usage seems almost right but ugly - should use dup2 instead
Comment: @AnttiHaapala You are right about the warning, I could have figured out the problem if I took a look at that. I am not sure what do you mean by "dup was almost right but ugly", could you share some more info?
Comment: @AnttiHaapala Indeed, this code is not robust against that case, but unless the asker is running the program in a strange way, that should not prevent it from working when s/he runs it.
Comment: 1) you should not even try to close 1 and 0. 2) your use of dup function and pipe output isn't right. Better read the Manual pages for complete explanations, but the idea is to create a "pipe" that your program will use with pipe (). The parent of the fork will write in 1 side of this pipe, the child will read on the other side. You then need to dup 1 or 0 of the entrance / exit of the pipe (depending). Finally don't forget to close the part of the pipe you are not using (only in the parent / child separated code, do not close both in the main part or nothing will work B-))
Comment: In any case, your compiler should have produced diagnostics messages! If not then time to up warning level!
Comment: Well close(1) followed by dup could have changed fd 0 instead if 0 wasn't open. That's why you should always use dup2 if you care about the resulting number
Comment: You aren't closing enough file descriptors in the processes. **Rule of thumb**: If you [`dup2()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/dup2.html) one end of a pipe to standard input or standard output, close both of the original file descriptors from `pipe()` as soon as possible. In particular, that means before using any of the [`exec*()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/execvp.html) family of functions. The rule also applies with either `dup()` or [`fcntl()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcntl.html) with `F_DUPFD`.
Comment: @Angevil Closing file descriptor 0 and 1 is legal.
Comment: Programs are entitled to assume that each of standard input, standard output and standard error are open — and on Unix, that means that each of file descriptors 0, 1, and 2 are open.  The program works correctly if it is given the environment it is entitled to assume it is given.  If it is launched with file descriptors 0 and 1 closed, it would work correctly if `dup2()` was used because `pfds[0]` would contain `0` and `pfds[1]` would contain 1, and `dup2(pfds[0], 0)` would not close `0`, and `dup2(pfds[1], 1)` would not close `1`.  Etc.  You have to try quite hard to break it with `dup2()`.
Comment: regarding the calculation of `k` (both of them)  the `k` should be an array of pointers to strings.  So rather than: `char *k = {"echo", "one", "two", "three", NULL};`  Suggest:  `char *k[] = {"echo", "one", "two", "three", NULL};`  Notice that `k` is now an array of pointers to strings
Answer: The issue is that you're assigning an array of strings to a char*. Both ks should be declared char* k[] = …. If your compiler didn't warn you about this, you need to enable more warnings.
Contrary to the comment, you're using close and dup correctly (but dup2 would be better).
Comment: Hmmm…you're probably correct that my alternative scenario would not create the problem I claimed — but even if I didn't diagnose a valid scenario (I was trying to keep it simple, and missed), nevertheless, in more complex pipelines, not closing both pipe file descriptors can lead to problems.  It is best not to leave file descriptors open that should not be open.  In some scenarios, it can be a security risk, too.
Comment: The calls to `close()` and `dup()` are, as you say, correct in themselves, and using `dup2()` would also be better.  However, as I've noted in a [comment](https://stackoverflow.com/questions/57245103/what-is-the-correct-step-to-execute-execve#comment100992341_57245103), the other dup'd file descriptor should be closed too.  This would be noticeable if instead of executing `echo`, the first command read its standard input; it wouldn't terminate because it would not get EOF on its standard input, because it has the write end of the pipe open, even though it'll never write to it.
Comment: @JonathanLeffler: Isn't it closing that descriptor (`dup(pfds[1]); close(pfds[0]);`), and the issue is that there's an unnecessary descriptor hanging around? Or am I misunderstanding?
Comment: No; the `dup(pfds[1])` does not close `pfds[1]`, and neither does `close(pfds[0])` — so `pfds[1]` (probably file descriptor 4) is still open in the process when the exec occurs.  It isn't marked `O_CLOEXEC` (close on exec) so the `exec` doesn't close it either.  (No `F_DUPFD_CLOEXEC` or `FD_CLOEXEC` either…)
Comment: @JonathanLeffler: "if instead of executing echo, the first command read its standard input; it wouldn't terminate because it would not get EOF on its standard input" – its stdin hasn't changed, has it?


Commuting operators in polar decomposition of normal operator

Question: Let $V$ be a finite-dimensional vector space and $f$ be a normal operator on $V$. Let $f=PU$ be polar decomposition of $f$, where $P$-nonnegative, self-adjoint and $U$ - unitary(orthogonal) operator. Prove that $PU=UP$.
My attempt: Since $f$ is normal then $ff^*=f^*f$ and $f^*=U^*P$ then $ff^*=PUU^*P=P^2$ and $f^*f=U^*PPU=U^*P^2U.$ Combining them we get that $$P^2=U^*P^2U \Leftrightarrow UP^2=P^2U$$
I don't know what to do next.
Can anyone show how to prove this, please?
Answer: You did most of the work. Because $P^2 = U^*P^2U$ and non-negative operators have unique non-negative square roots, we can conclude that $P = U^*PU \iff UP = PU$.
Equivalently, we could use the fact that there exists a polynomial $g(x)$ for which $g(P^2) = P$.
Comment: @ZFR Taking $y = U(x)$, we see that $(PU(x),U(x)) = (Py,y)$. Because $P$ is non-negative, $(P(y),y) \geq 0$.
Comment: Every time I am very close to the solution but never can solve it :)  To be honest I did not get how did you get that $g(P^2)=P$?
Comment: @ZFR Either this fact or the uniqueness of square roots is usually proven as a theorem in textbooks/notes. It's non-trivial; the usual proof uses the spectral theorem
Comment: Let me ask you the question regarding the first paragraph of your answer: So we know that $P^2=(U^*PU)^2=U^*P^2U$. And we see that $U^*P^2U$ is self-adjoint and non-negative. I know that it has unique self-adjoint non-negative square root. Also $P$ is self-adjoint and non-negative. We have to show that $U^*PU$ is self-adjoint and non-negative. Self-adjointness is very obvious. But non-negativity is not. We have to consider $(U^*PU(x),x)=(PU(x),U(x))$
Comment: Ohh god! I am so stupid. Thanks a lot! I did not notice this triviality at all!


Oracle job to run daily in Oracle Express edition 11g

Question: ![enter image description here][1]
I just updated with a new version of the database. I've previously used Oracle database versioyon 10. Now I installed Oracle 11g Express edition.
I had the three jobs on the DBMS_jobs.
I programmed it to run them at a particular time on a daily basis. every day at 01:00 in the stored procedure was running on a regular basis.
begin
  sys.dbms_job.submit(job => :job,
  what => 'BEGIN CALL_MY_SP; END;',
  next_date => to_date('24-09-2013 01:00:00', 'dd-mm-yyyy hh24:mi:ss'),
   interval => 'TRUNC(SYSDATE + 1) + 1/24');
  commit;
end;

Although the same job to the new Oracle database running. But this job is manually executing.
How do I run automatically on a regular basis every day? But What if I did not!
https://i.stack.imgur.com/rePVq.png
your job
http://l1309.hizliresim.com/1f/s/sz6pg.png
Answer: To run manually, try:
exec dbms_scheduler.run_job('MY_JOB',false);

and to stop manually, use:
exec dbms_scheduler.stop_job('MY_JOB',false);

If your question is how to scheduler a job in Oracle, its a big topic (see Documentation ), but a simple example would be:
BEGIN

  dbms_scheduler.create_job(
    job_name=>'JOB_TEST2',
    job_type=>'PLSQL_BLOCK',
    job_action=>'BEGIN
    dbms_lock.sleep(10);
    END;',
    start_date=>systimestamp,
    repeat_interval=>'FREQ=MINUTELY;INTERVAL=1',
    number_of_arguments=>0,
    enabled=>TRUE,
    comments=>'Job duration is 10 seconds, launches every minute'
  );

END;

Since it was created as enabled, it will start running automatically.  You can verify this looking at the logs (dba_scheduler_job_run_details) and can view basic attributes via dba_scheduler_jobs
Comment: Maybe I misunderstand your question, but your post said: "But I'm trying to manually executing. How do I run this job on oracle oracle 11g express edition?"
Comment: I would like to run automatically on a regular basis
Comment: btw @ant , you should use dbms_scheduler for scheduled jobs.  But again, perhaps I misunderstand your question
Comment: @ant what error exactly?  did you create a scheduler job like my example?
Comment: yes... i drop my job after run above your job . i change only this 'dbms_lock.sleep(10);' to CALL_MY_SP; .
Comment: I have described the first time I removed all the job.  after run above your job... (only change this line  dbms_lock.sleep(10); to SP_GUNCETELEME;.... and finally I'm trying to run the job manually. I get the error above. No other error. job work also be automatically http://l1309.hizliresim.com/1f/s/sz6pg.png
Comment: @ant I just needed the ora code (27492).  There are several possible reasons, see [this post](http://stackoverflow.com/questions/18898444/oracle-enterprise-manager-job-is-hung) for more info.
Comment: solved the problem, this leads to. `alter system set job_queue_processes = 10;`


How To Transmit Data Between A Webpage And An Android-App

Question: I need an approach: My aim is to transmit about every minute a number from an Android-App to Webpage, opened on a Computer. It needs to be fast, the computer just has Firefox installed (I can't install programs) and the devices aren't in the same network (has to be over Internet, not bluetooth or similar).
My current solution is a MySQL-Bridge: The opened Webpage on the computer retrieves data from the SQL-Database, filled by the android device, about every second (dirty solution), the problem is, that I use PHP and a while(true)-loop. The webhoster does not allow to set the time limit and I get after a few seconds an error. I have also tried a socket, but then the page loaded the whole time. Have I made a mistake there?
What approach is the best one for my project?
Thanks
Comment: You may like to have a look at 'rethinkdb' and use it instead of mysql. They probably have inbuilt web-sockets support which can update the webpage.
Comment: Rather than the while loop, you could try setting up a `cron` job. Most webhosts allow this - [here is BlueHost's help](https://my.bluehost.com/cgi/help/411) - and it would get you around the process timeout problem. Furthermore, use PHP's [`sleep(1)`](https://php.net/manual/en/function.sleep.php) rather than a `while(true)`! Note this is getting off-topic for here; follow-up on perhaps https://serverfault.com/, please.
Comment: Oh, and for a quick-and-dirty solution, I personally would use Parse.com and JavaScript.
Answer: If you are right about the execution timeout of your php scripts then are there no options you could do. If not, you could try to run the command set_time_limit(0);, this removes the limit of the script execution. If this works you could try so use a XMPP like protocol. I'm sure there are some ready to use libraries.


Nginx appears to be unable to connect to any upstream servers

Question: We have an Nginx instance serving as a reverse proxy for a couple of domains, each hosted on their own box. Or, well, we used to. Yesterday I had barely finished updating the Nginx box with Pacman when apparently the UPC failed and the whole system lost power. When the UPC was swapped out, the system turned back on and everything worked perfectly - except Nginx.
After trying various permutations of the config file, it seems as though it's lost the ability to contact other servers, but I can't definitively confirm this because all the log says is worker process nnn exited on signal 11 (core dumped) and, though I've added in the lines to tell it to store the dumps, none are actually being produced. It's able to serve files stored locally, and I'm able to curl the other servers from the command line, but when I try accessing any domain with a proxy_pass entry it craps out and gives an empty response.
The install is now version 1.6.3, though I believe it was that before the update too.
Edit: Output of nginx -V:
nginx version: nginx/1.6.3
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-imap --with-imap_ssl_module --with-ipv6 --with-pcre-jit --with-file-aio --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_realip_module --with-http_spdy_module --with-http_ssl_module --with-http_stub_status_module --with-http_addition_module --with http_degradation_module --with-http_flv_module --with-http_mp4_module --with-http_secure_link_module --with-http_sub_module

Comment: What modules are you using?  Does nginx start up without any errors? Can you post the output of "nginx -V"?
Comment: @ChadSmith I have logging enabled all the way down to the `info` level, and the only indication I can see of anything unusual is the above mentioned `signal 11` mesages.
Comment: Which linux distro are you using?  This still seems to me like a dependency has failed.
Comment: One last comment, have you made sure that your temp directories still exist and have the correct permissions after your crash?  The actual error thrown by nginx may have nothing to do with what its logging.
Comment: @ChadSmith When you asked me to grab the list of modules I noticed that the proxy module wasn't on there, which led me to the solution. Thanks for the help, however indirect. ;)
Answer: I just ran pacman -Syu again and there was another update for it. Apparently the package had accidentally been compiled without proxy support and I didn't notice.


Internal Server Error when running Flask App

Question: Really appreciate your help! I am a beginner to Flask, playing with it to build out APIs.
When executing the below code snippet and run the code in vs-code terminal, it appears to run the app on http://127.0.0.1:5000/.
However, when I click-through the URL within the terminal - to launch the URL in the browser and expect to display 'Hello World' - it displays 'Internal Server Error' on the page. It then prints 'ERROR in app: Exception on / [GET].
Code:
from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    print('Hello World!')

app.run(port=5000)

Terminal:
WKMGB0671549:REST-APIs josshepp$ python3 app_copy.py
 * Serving Flask app "app_copy" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
Hello World!
[2019-07-11 08:51:50,920] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    return self.finalize_request(rv)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/flask/app.py", line 1967, in finalize_request
    response = self.make_response(rv)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/flask/app.py", line 2097, in make_response
    "The view function did not return a valid response. The"
TypeError: The view function did not return a valid response. The function either returned None or ended without a return statement.
127.0.0.1 - - [11/Jul/2019 08:51:50] "GET / HTTP/1.1" 500 -

Comment: "TypeError: The view function did not return a valid response. The function either returned None or ended without a return statement."
Answer: Your view function does not return a valid response. In short, return "Hello World!" instead of printing it:
@app.route('/')
def home():
    return 'Hello World!'

Flask automatically converts several types of return vales into Response for you, but you can read up on how they convert (link):

The return value from a view function is automatically converted into a response object for you. If the return value is a string it's converted into a response object with the string as response body, a 200 OK status code and a text/html mimetype.

If the term view function is confusing, it is described by flask as (link):

A view function is the code you write to respond to requests to your application. Flask uses patterns to match the incoming request URL to the view that should handle it. The view returns data that Flask turns into an outgoing response.
Comment: No problem @Larry125! If you liked the answer please also use votes and accepted answer to indicate it.


Calculate new graph point with coordinate and angle

Question: On a Cartesian graph I have a point at $x = 0$ and $y = 0$. This point needs to move forward at a $30^\circ$ angle. It should travel forward at this angle for $1.75$ on the graph. What equation can I use to do this?
Summary:

Cartesian graph point $(0, 0)$
Moves forward at a $30^\circ$ angle
Needs to travel forward at the angle $1.75$ on the graph
Is there an equation for this?
Comment: I suggest you read up on [trigonometry](http://en.wikipedia.org/wiki/Trigonometry)
Comment: Thanks captain obvious. But I'm just getting back into math after a very long time.
Comment: Looks like (sin( angle * pie / 180) * 1.75) worked.
Answer: There is an way to determine the coordinates of the point. I assume that you mean "moves $1.75$ units along a line in the first quadrant that makes a $30$ degree angle with the positive x-axis", or something like that.
Draw a reference right triangle
This one is pretty sloppy... learn "SOHCAHTOA", a mnemonic device for trig ratios. Then we can say
$$\cos(30) = x/1.75  $$
$$\sin(30) = y/1.75$$
And solve for x, y.
Comment: your missing (angle * pie / 180) * 175