Ways to increase forecast accuracy

Question: Situation

Our use case:

demand forecasting for sales and operations planning
monthly granularity, ~5 years worth of historical data available
goal is to forecast future time windows of 1, 3 and 12 months
forecasts are generated twice per month

We have a time series forecasting system in place. It uses basic ARIMA and Exponential Smoothing techniques.

Problem:

The forecasts are deemed not accurate enough for the business stakeholders.
We're currently evaluating different ways to improve forecast accuracy.

(I'm not going into details around baseline accuracy, target accuracy etc here. At this point, we're looking for general approaches for improving accuracy that we could explore further)
Current Approaches
We're currently thinking along three dimensions:

Tweaking the algorithms

advanced non-ML methods: we plan to run tests using Auto ML or frameworks like Prophet, sktime to get an indication of how well more sophisticated algorithms perform
use machine learning in general: we are thinking about using ML (regression trees, gradient boosting) which requires rewriting our preprocessing pipeline

Enriching the data

include external data (and use multivariate methods): There are a couple of hypotheses of macroeconomic time series that could inform demand forecasts. We'd have to acquire datasets and test those, however.
preprocessing: we ran basic checks for outliers, missing values etc. our data seems pretty good. Maybe there are more elaborate preprocessing steps (smoothing, transforms etc.) that work well in practice

Feasibility

this is linked to point 1.: based on our algorithm review we might "verify" (in the loosest sense) that we can only forecast up to some level of accuracy and resort to e.g. only forecasting trends

Question
Which methods can be applied to increase forecast accuracy?
Thanks!
Comment: The best practical approach for the firm (its owners/shareholders/board of directors) is to fire all those **business stakeholders** who believe you can predict the future with a huge degree of uncertainty better than God, because this prediction is the most sophisticated thing that can be done in the business dimension within a company and it cannot be supervised or guided by amateur idiots with MBA diplomas and polished business résumés.
Comment: Thank you for not going into target accuracy. As long as you don't know how forecastable your data is, setting targets is an exercise in futility. And please don't use published benchmarks: [What is the acceptable level of accuracy while doing Weekly Time Series Forecast](https://stats.stackexchange.com/q/274317/1352), [Is there any standard / criteria of good forecast measured by SMAPE and MASE?](https://stats.stackexchange.com/q/473716/1352), [Acceptable Standard for MAPE](https://stats.stackexchange.com/q/435074/1352), all applicable to all other error metrics.
Comment: just as a comment as there is a wide response already, the best way is to understand the error metric you use and the validation method you use as every method is checking different things and is valid for different scenarios. The wrong thing and very common is to change the metric and method as forcing your forecasting model to be "good".
Comment: Domain knowledge?
Answer: I have been forecasting retail demand for 16 years now. Retail is probably not what you are interested in, but a few comments on your ideas plus a few other ideas might be helpful.

Tweaking the algorithms: to be honest, I usually find that better algorithms are always beaten by better data, and better understood data. More complex methods will often give better results. (Often, not always. In the recent M5 forecasting competition, a trivial benchmark beat 92.5% of the submissions at the lowest granularity, see Kolassa, 2022.) What is often useful is thinking about what the forecasting method should be capable of doing. If you have important causal drivers, you should use a method that can use them, so not plain vanilla Exponential Smoothing or ARIMA - but a simple regression will often be quite competitive with a highly complex DL network, at a fraction of the cost and headache.

Enriching the data: if you use external drivers, remember that you will need to forecast these themselves in production, and forecasts of macroeconomic and many other series is highly imprecise! It's easy to fall into a trap here, using actual future values in testing such predictors and thus overestimate your certainty and accuracy improvement you will get in production. And as above, the question is whether any improvement is worth the added cost and complexity of acquiring data and feeding it into the pipeline (and maintaining all this).
I usually find that cleansing the data you do have, and understanding drivers, is much more important. Are there no sales because of supply chain problems during some months? Mark these and ignore them in training (if your method will allow so). If this happened, did demand switch to substitute products? If so, mark these periods on the substitutes, because when the original product comes back online, the substitutes will presumably see a drop in demand. Do you run promotions or similar activities? Model these. Understanding your data and cleaning it is always more important than trying more complex models.

Feasibility: this is the elephant in the forecasting room. You can't forecast a flipped coin with more than 50% accuracy, and if your business stakeholders "require" more accuracy than you can achieve, they have a problem. See How to know that your machine learning problem is hopeless?.

A few other thoughts:

Making processes more forecastable: some activities make life actively harder on the forecasters (and on the rest of the supply chain, too). Promotions in retail are notoriously hard to predict, and have cannibalization impacts on other products, and on the focal product after the promotion. There is a reason why well performing retailers like Walmart and dm drogeriemarkt in Germany run Every Day Low Price strategies - it's just easier on the supply chain, and makes forecasting easier, too. Relatedly, there have been spats between Consumer Packaged Goods manufacturers and retailers, which went as far as the manufacturers stopping deliveries. This will make forecasting harder down the line for everyone involved. Similar issues arise from product proliferation; it's easier to forecast if we have five flavors of yogurt than if we have thirty, half of which are listed and delisted all the time, even if your marketing department loves new product introductions. No, I'm not saying the forecaster has the clout to change business processes. But it might be worthwhile sitting down with other people and figuring out how their activities are negatively impacting the forecasting function.

Mitigation. Imprecise forecasts can be mitigated through safety stocks. No, nobody likes these, but once we have reached the end of our tether in terms of forecast accuracy, we can buffer the impact.

Reducing the role of forecasting: relatedly, we can reduce the reliance on forecasts by pushing customization down the line as far as possible: if we paint our widgets only right before they are shipped out, we may only need to forecast "total widgets", rather than "red widgets", "yellow widgets" and "light pink-mauve widgets" separately (which will be harder).

Measure the costs of bad forecasts and accuracy improvements: per above, often there is a point of diminishing returns in forecast improvements, where you can start spending serious money to only get a small accuracy improvement. It's worthwhile to figure out how much your accuracy improvement is worth in currency terms. I give a couple of examples in Kolassa (2022), and issue 68 of Foresight is devoted to this topic (full disclosure: I'm a Deputy Editor at Foresight). Essentially, if your logistical constraints and economic batch sizes are "large", then even better forecasts may lead to the exact same business and production decisions.

Accuracy measures: there is a huge and embarrassing disconnect between forecast accuracy measures and business relevance. Many people like the Mean Absolute Percentage Error, because it looks so easy to interpret. It isn't, it can be highly misleading, it can easily be gamed, and I have never seen a business process that would profit from a forecast only because its MAPE is lower. (If your bonus depends on the MAPE and you are cynical, you can simply game it.) The MSE and scaled variants at least elicit unbiased expectation forecasts, but in a world of safety stocks, surprisingly few business processes really leverage expectation forecasts. The relationship between quantile losses and safety stocks/service measures is a little better, but quantile forecasting is often underappreciated. So I would seriously recommend you look at your error measures and figure out whether they are useful for your business processes.

Hierarchical forecasting: you may be able to leverage hierarchies, whether in the product, the location or the time dimension. There has been a lot of work on optimal reconciliation, and it typically improves forecasts across the board. However, most of the work here up to very recently has only been on expectation forecasting, and per the previous point, quantile forecasts are often much more relevant.

Talk to experts: forecasting is a science, and there are experts out there, many of whom will be happy to talk to you, some even for free. I have been involved with the IIF and its publications, notably Foresight, there is also the Institute of Business Forecasters, and depending on where you are in the world, you might want to reach out to institutions like the Centre for Marketing Analytics and Forecasting at Lancaster University Management School. They regularly offer to have their M.Sc. students do a thesis in a company, and such students might be a reasonably cheap source of new ideas. (Full disclosure again: I'm affiliated with the CMAF.)
Comment: "[T]here are [forecasting] experts out there, many of whom will be happy to talk to you, some even for free."  I agree.  It does seem like there are [some](https://stats.stackexchange.com/users/1352/stephan-kolassa) forecasting experts who are willing to advise @movingabout for free...
Answer: First of all I agree 100% with Stephen's answer, I'll just add a little bit from my 2 years of experience!

The ML vs traditional methods IMO boils down to a simple question:
Do you have good drivers to use as variables?

Time series methods work best for time series, of course you can use other factors to aid but with 1 time series going to 1 model you also need to be careful with those features. ML (boosted trees / RFs like you suggest) work best for tabular data where you tend to lose your time series structure so you have to make up for that with good tabular features and simply 'represent' time with other features.
Things like price of products, marketing expense etc. If you don't have these types of variables for your domain then I would bet a decent stat engine outperforms a state-of-the-art ML model in a production setting. That production setting piece is important, with an ML model you have very little control of the actual forecast -you get what you get. A stat engine should allow you to on-the-fly switch to another method if the forecast of the current one is wonky, which leads to my next thought. Just remember though, if you use something like GDP you then probably have to forecast for GDP to use it in the future which is probably very problematic! Or use lagged GDP which may not be as useful.

What makes a decent stat engine?

Your model portfolio (what you are looking into now) is important but model selection and a business logic layer is everything. For model selection look to time series cross validation. For the business logic layer I would lean on the stakeholders of the forecast. For example, you probably want to assign a 'demand type' for each given time series. Like if 30% or more of the series is 0 then you want to assign it a 'type' which would only allow certain models to be selected such as simple Exp smoothing or croston or mean. An arima may product wonky results in those settings. You could also check to ensure the forecast doesn't go from 5 units to 50 million, something that is possible in an overparameterized arima. You could check to see if there a certain product lifecycles at play, like if there is a build up and fall off over the years and then fit a more local model or weigh the more recent years more if your method takes sample weights. A lot of possibilities here for adding logic that aids the engine.
In summary,
Add some naive methods, you could add some other methods but I personally would stay away from Prophet - autoarima + autoets + naive methods (mean, last period, last seasonal period) will be a good start, take a look at your model selection criteria to ensure it is robust, add some 'logic' to help ensure that the model is appropriate and isn't just merely the one that minimizes some loss function.
But most importantly -
Look at your forecasts.
Setup some quick flags to surface forecasts where the model suggests new maxes/mins or the average of the forecast period is significantly different than the average of the history. Figure out if there are commonalities between these flagged series like a ton of zeros etc. Many times it is just an odd bug in your code meaning that your outlier detection isn't working right or some other issue that causes bad results.
If you have done all of that and want additional models to try my main recommendations would be:

Theta - there are tons of implementations across python and r. Theta plus auto arima do well in general.
Croston - pretty standard for intermittent data.

A lot of 'AutoML' time series methods literally try everything under the sun, take a lot of time, and don't add too much value beyond all the methods listed above.
Additionally you could try out some of my personal projects in the field

ThymeBoost which is just gradient boosted time series decomposition with traditional methods like ETS and ARIMA
TimeMurmur my newest which does large scale LightGBM time series forecasting, probably wouldn't use it in prod but you could give it a shot as a baseline.

Ways to increase forecast accuracy


PyArrow: How to copy files from local to remote using new filesystem interface?

Question: Could somebody give me a hint on how can I copy a file form a local filesystem to a HDFS filesystem using PyArrow's new filesystem interface (i.e. upload, copyFromLocal)?
I have read the documentation back-and-forth, and tried a few things out (using copy_file() with FS URIs), but none of it seems to work. The usage of the legacy HDFS API is straightforward, but it is deprecated, though the new API seems to be incomplete. Of course moving chunks of data among file descriptors is a solutions, but why copy_file() exists then?
Answer: There are no functions in the new (or old) filesystem APIs for transferring files between filesystems.

Of course moving chunks of data among file descriptors is a solutions

I'm not sure if this is what you were thinking, but here is a simple utility (and demo) on how to do this from python:
import filecmp
import pyarrow.fs as pafs

BATCH_SIZE = 1024 * 1024

def transfer_file(in_fs, in_path, out_fs, out_path):
    with in_fs.open_input_stream(in_path) as in_file:
        with out_fs.open_output_stream(out_path) as out_file:
            while True:
                buf = in_file.read(BATCH_SIZE)
                if buf:
                    out_file.write(buf)
                else:
                    break

local_fs = pafs.LocalFileSystem()
s3fs = pafs.S3FileSystem()
in_path = '/tmp/in.data'
out_path = 'mybucket/test.data'
back_out_path = '/tmp/in_copy.data'

transfer_file(local_fs, in_path, s3fs, out_path)
transfer_file(s3fs, out_path, local_fs, back_out_path)

files_match = filecmp.cmp(in_path, back_out_path)
print(f'Files Match: {files_match}')

I would expect transfer_file to get good performance.  There may be some situations (e.g. reading from S3) that could benefit from a parallel read using read_at which would require a bit more complexity but should also be doable.

but why copy_file() exists then?

copy_file copies a file from one name on a filesystem to a different name on the same filesystem.  It cannot be used to copy files between different filesystems.
Comment: Makes sense! I'll wait a few days for more answers, but your thoughts are confirming my assumptions. Thank you!
Comment: This definitely solved my problem. Thank you Pace!
Answer: To add to @Pace's answer (too long for a comment):  I was copying gzip files (*.gz) which pyarrow was (by default) decompressing with each read() call and then compressing again with each write() call.  I verified this with the print statement shown below in Pace's version of the code.
So, to get significantly faster transfers/copies, turn off compression:
def transfer_file(in_fs, in_path, out_fs, out_path):
    with in_fs.open_input_stream(in_path, compression=None) as in_file:
        with out_fs.open_output_stream(out_path, compression=None) as out_file:
            while True:
                buf = in_file.read(BATCH_SIZE)
                if buf:
                    print(f'buf size: len(buf)')
                    out_file.write(buf)
                else:
                    break


why does the same variable have a different slope when incorporated into a linear model with multiple x variables

Question: When I call for the summary of my linear model it shows X2 to have a negative slope. But
when i call the same variable in a linear model of its own it has a positive slope. why is it not just the same slope for x2 in both models?
 summary(lm(y~x1+x2+x3+x4))

Call:
lm(formula = y ~ x1 + x2 + x3 + x4)

Residuals:
     Min       1Q   Median       3Q      Max
-0.31054 -0.07664 -0.00567  0.06249  0.35183

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.05794    0.02342   2.473  0.01387 *
x1           0.25579    0.04514   5.666 3.09e-08 ***
x2*         -0.02356*   0.04781  -0.493  0.62248
x3           0.45445    0.04878   9.317  < 2e-16
x4           0.13801    0.04620   2.987  0.00302

> lm(y~x2)

Call:
lm(formula = y ~ x2)

Coefficients:
(Intercept)           x2
     0.2607       **0.2110**

Comment: (+1) for this question. To me, it seems like it has been handled before (the question is specifically about the intercept, but it works the same for the slope). Does this answer your question? [Why is the intercept in multiple regression changing when including/excluding regressors?](https://stats.stackexchange.com/questions/429526/why-is-the-intercept-in-multiple-regression-changing-when-including-excluding-re)
Comment: Also related Related: [Why and how does adding an interaction term affects the confidence interval of a main effect?](https://stats.stackexchange.com/questions/364711) and [Why is the intercept changing in a logistic regression when all predictors are standardized?](https://stats.stackexchange.com/questions/429623)
Answer: The following viewpoint may help your intuition:

Let some there be some data distributed according to a quadratic curve:
$$y \sim \mathcal{N}(\mu = a+bx+cx^2, \sigma^2 = 10^{-3})$$
For instance with $x \sim \mathcal{U}(0,1)$ and $a=0.2$, $b=0$ and $c=1$. Then a linear curve and a polynomial curve will have very different coefficients for the linear term.

set.seed(1)
x <- runif(100, 0,1)
y <- rnorm(100, mean = 0.2+0*x+1*x^2,
                sd = 10^-1.5)
plot(x,y, ylim = c(0,1.5),
     pch = 21, col = 1 , bg = 1, cex = 0.7)

mod1 <- lm(y~x)
mod2 <- lm(y~poly(x,2, raw =TRUE))

xs <- seq(0,10,0.01)
lines(xs,predict(mod1,newdata = list(x = xs)), lty = 2)
lines(xs,predict(mod2,newdata = list(x = xs)),lty =1)

legend(0,1.5,c("y = 0.009 + 1.023 x", "y = 0.193 + 0.016 x + 0.994 x^2"), lty = c(2,1))

Correlation
The reason is that the variables/regressors $x$ and $x^2$ correlate.
The coefficient estimates computed with a linear regression are not a simple correlation (perpendicular projection onto each regressor seperately):
$$\hat{\beta} \neq \alpha = \mathbf{X^t} y$$
(this would give coefficients $\alpha_1$ and $\alpha_2$ in the image below, and these coordinates/coefficients/correlations do not change when you add or remove other regressors)
Using the correlation/projection $\mathbf{X^t}y$ is wrong, because if there is a correlation between the vectors in $\mathbf{X}$, then there will be an overlap between some vectors. This part that overlaps will be redundant and added too much. The predicted value $\hat{y} = \alpha \mathbf{X}$ would be too large.
For this reason there is a correction with a term $(\mathbf{X^t}\mathbf{X})^{-1}$ that accounts for the overlap/correlation between the regressors. This might be clear in the image below which stems from this question: Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

Intuitive view
So the regressors $x$ and $x^2$ both correlate with the data $y$ and they both will be able to express the variation in the dependent data. But when we use them together then we are not gonna add them according to their single independent effects (according to correlation with $y$) because that would be too much.
If we use both $x$ and $x^2$ in the regression then obviously the coefficient for the linear term $x$ should be very small since this is the same in the true relation.
However, when we are not using the quadratic term $x^2$ in the regression (or otherwise add a bias to the coefficient for the quadratic term), then the coefficient for $x$ which correlates somewhat with $x^2$ will partly take correct this (take over) and... the value of the estimate for the coefficient of the linear term will change.
See also:

regression with multiple independent variables vs multiple regressions with one independent variable
Why is the intercept in multiple regression changing when including/excluding regressors?
Why and how does adding an interaction term affects the confidence interval of a main effect?
Why is the intercept changing in a logistic regression when all predictors are standardized?
Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression
Does adding more variables into a multivariable regression change coefficients of existing variables?
Estimating $b_1 x_1+b_2 x_2$ instead of $b_1 x_1+b_2 x_2+b_3x_3$
Why do regression coefficients change when excluding variables?
Does the order of explanatory variables matter when calculating their regression coefficients?
Comment: Sextus Empiricus, thank you for your detailed explanation and helpful reading  suggestions..
Answer: You can think of the other variables regression with all the variables as something you are controlling for.
Let's say you are estimating weight gain, with the weight of the food eaten as a predictor. You see than you lose weight when you eat more, i.e. the slope is negative, when it should be the opposite by common sense. You realize that days when you eat more as measured by the weight of the food, you have eaten more vegetables (that have very little calories) since they don't make you full as fast as the more calorie-rich foods.
So then you add a predictor of the amount of calories in the food you eat. You'll see that the weight of the food most likely turns insignificant (as in your case the predictor x2) and the slope can change the direction, since it is actually the calories which make you gain weight, not the weight of the food.
This is closely related to the Simpson's Paradox, where missing variables would lead to the opposite conclusion instead of the correct one. In your case, it is most likely due to multicollinearity. In my example the weight of the food and the calories were negatively correlated. You can check for multicollinearity using the variance inflation factor (VIF) like this: car::vif(lm(y~x1+x2+x3+x4))
Comment: Yes, my goal is to make predictions. I have concluded that the radical change indicates that X2 is a reciprocal suppression variable. Would you agree with this estimation?
Comment: kkz, thanks for your explination. I have run the vif and am trying to interpret the results.(below). the vif for x2 seems low, just 1.215179. Does this mean that there might be somethingelse going on in my data?                                                                                > car::vif(lm(y~x1+x2+x3+x4))
      x1       x2       x3       x4
1.083421 1.215179 1.264715 1.134739
Comment: Yes, then it's something else. If your goal is to make accurate predictions instead of inference, then the model with highest accuracy on the test set should be used and you don't have to worry about the flip of the sign. If your goal is inference then you could still say that the sign changes because you control for the other variables.


How do I put a MultiDatesPicker in Spanish?

Question: In the view of my Asp.Net App I have the following code at the end:

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jquery-ui-multidatespicker/1.6.6/jquery-ui.multidatespicker.css" />

That is the script which makes a MultiDatePicket appear in my view (https://dubrox.github.io/Multiple-Dates-Picker-for-jQuery-UI/)
And I call the MultiDatepicket in a Div like this:

      <input id="mdp-demo" autocomplete="off" name="selectedDates">

Now my question is, how do I get the Calendar which is seen in the view to be a Spanish Calendar instead of the default English Calendar?
According to a tutorial I found, you can manually modify a normal DatePicket so the result in the view is a calendar in Spanish, with a code like this:

But I can't find how to make this work for a MultiDatePicket, how can I do it? How do I include the code which modifies the name of the days/months in the MultiDatePitcket script to change the calendar to a Spanish Calendar?
Answer: Make sure you setup your regional configuration before you call $.fn.multiDatesPicker.
The order your scripts load matters, because they load top-down.

$.datepicker.regional['es'] = {
  closeText: 'Cerrar',
  prevText: '< Ant',
  nextText: 'Sig >',
  currentText: 'Hoy',
  monthNames: ['Enero', 'Febrero', 'Marzo', 'Abril', 'Mayo', 'Junio', 'Julio', 'Agosto', 'Septiembre', 'Octubre', 'Noviembre', 'Diciembre'],
  monthNamesShort: ['Ene', 'Feb', 'Mar', 'Abr', 'May', 'Jun', 'Jul', 'Ago', 'Sep', 'Oct', 'Nov', 'Dic'],
  dayNames: ['Domingo', 'Lunes', 'Martes', 'Miércoles', 'Jueves', 'Viernes', 'Sábado'],
  dayNamesShort: ['Dom', 'Lun', 'Mar', 'Mié', 'Juv', 'Vie', 'Sáb'],
  dayNamesMin: ['Do', 'Lu', 'Ma', 'Mi', 'Ju', 'Vi', 'Sá'],
  weekHeader: 'Sm',
  dateFormat: 'dd/mm/yy',
  firstDay: 1,
  isRTL: false,
  showMonthAfterYear: false,
  yearSuffix: ''
};

$.datepicker.setDefaults($.datepicker.regional['es']);

$('#mdp-demo').multiDatesPicker({ dateFormat: "dd/mm/yy", });

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jquery-ui-multidatespicker/1.6.6/jquery-ui.multidatespicker.css" />

  <input id="mdp-demo" autocomplete="off" name="selectedDates">


for yii xupload extensition

Question: i wants to using a multiple attribute in xupload extension
my code :-
Yii::import("xupload.models.XUploadForm");

  $this->widget( 'xupload.XUpload', array(
            'id'=>'uploadwidget',
            'url' => Yii::app( )->createUrl("$mo"),
            'model' => $photo,
            'htmlOptions' => array('id'=>'testmagazines-form'),
            'attribute' => 'file',
            'multiple' => true,
            'options'=>array(
                              'maxNumberOfFiles'=>3,
                              'maxFileSize'=>"js:$('#filesize').val()",
                              'acceptFileTypes'=>"js:/(\.|\/)(jpeg)$/i",

                              'completed' =>"js:function(id, fileName, responseJSON){
                                var value = $('#number').val();
                                var value1 =  parseInt(value) + parseInt(1);
                                $('#number').val(value1);
                                jQuery(document).trigger('close.facebox');
                                }",
                                'success'=>"js:function(res){
                                 var b = $('#number').val();
                                if(b=='1'){
                                $('#filesize').val('104857600');

                                }else if(b=='2'){
                                $('#filesize').val('104857600');
                                }else if (b==3){
                                $('#filesize').val('10*1024*1024');
                                }
                                    alert(res[0].mode);
                                }",
                              'messages'=>array(
                                        'typeError'=>"{file} has invalid extension. Only {extensions} are allowed.",
                                        'sizeError'=>"{file} is too large, maximum file size is {sizeLimit}.",
                                        'minSizeError'=>"{file} is too small, minimum file size is {minSizeLimit}.",
                                        'emptyError'=>"{file} is empty, please select files again without it.",
                                        'onLeave'=>"The files are being uploaded, if you leave now the upload will be cancelled."
                                                ),
                                        'showMessage'=>"js:function(message){ alert(message); }"
                                        ),
            )
        );

in this i want to every upload file exe. may be difference so i how to handle it.
Answer: i got a answer for it:-

i can upload a multiple types of images with multiple accept file type and also size:
Solution:-
in find a jquery.fileupload-ui.js in a extension folder:-
then add a parameter in this js.
example :-
  pqueryid : undefined,
        pqueryid1 : undefined,

        lqueryid : undefined,

        lqueryid1 : undefined,

        queryid : undefined,

        queryid1 : undefined,

all Attribute i need at run time . so put a value in option parameters.
 add: function (e, data) {
                var that = $(this).data('fileupload'),
                    options = that.options,
                    files = data.files;

                $(this).fileupload('process', data).done(function () {
                    that._adjustMaxNumberOfFiles(-files.length);

             if(counter==2)
                {
                var dataurl ='/project/client/magazinecms/admin/index.php?r=testMagazines/uploadnew&imagename=dev_cert_path&id='+that.options.queryid1+'&queryid='+that.options.queryid+'&m=testMagazines'
                that.options.url = dataurl;
                that.options.acceptFileTypes =/(\.|\/)(pem)$/i;
                counter--;
                 }else if(counter==1)
                {
                var dataurl ='/project/client/magazinecms/admin/index.php?r=testMagazines/uploadnew&imagename=prod_cert_path&id='+that.options.pqueryid1+'&queryid='+that.options.pqueryid+'&m=testMagazines'
                that.options.url = dataurl;
                that.options.acceptFileTypes =/(\.|\/)(pem)$/i;
                counter--;
                }else{
                var dataurl ='/project/client/magazinecms/admin/index.php?r=testMagazines/uploadnew&imagename=logo_path&id='+that.options.lqueryid1+'&queryid='+that.options.lqueryid+'&m=testMagazines'
                that.options.url = dataurl;
                that.options.acceptFileTypes =/(\.|\/)(jpeg|jpg|png)$/i;
                counter--;
                }

                    data.isAdjusted = true;
                    data.files.valid = data.isValidated = that._validate(files);
                    data.context = that._renderUpload(files).data('data', data);
                    options.filesContainer[
                        options.prependFiles ? 'prepend' : 'append'
                    ](data.context);
                    that._renderPreviews(files, data.context);
                    that._forceReflow(data.context);
                    that._transition(data.context).done(
                        function () {
                            if ((that._trigger('added', e, data) !== false) &&
                                    (options.autoUpload || data.autoUpload) &&
                                    data.autoUpload !== false && data.isValidated) {
                                data.submit();
                            }
                        }
                    );
                });
            }

and making a function any where need.
manage it.

for yii xupload extensition


Translate view header

Question: I add a simple text as view header but it is not exists in translatable text in admin/config/regional/translate.
In Drupal 7 we could even translate unformulated views header and footer with the following steps:

Go to admin/config/regional/i18n/strings
Check "full HTML" in "Translatable text formats" section
Go to admin/config/regional/translate/i18n_string
Click on Refresh strings

Has this feature been removed form Drupal 8? (I don't think so.)
How can I translate view header and footer simple text?
Comment: Plain Drupal 7 doesn't have any admin/config/regional/i18n/strings; that is added by the i18n module.
Comment: @kiamlaluno I mean with module.drupal 7 by default not support i18n
Comment: There is a thread, that explain config translation:
http://drupal.stackexchange.com/questions/189872/how-to-translate-the-views-page-title
Answer:
Enable "Configuration Translation" core module
Go to your view edit and you will see a "Translate view" tab
You'll find your language in the list there. Then click on "Edit"
next to the language
Expand panels and you will find your header text under the "Header"
panel
Comment: 5 years ago( when I asked the question) this option wasn't there ;). thank you for your answer. could be helpful for other guys.
Answer: If my previous answer doesn't work fine. Try with this solution.

In view add new field "Custom text" (Click checkbox Exclude from display)
In "Custom text" add code like this

{% trans %}Contact us{% endtrans %}

{% trans %}Please contact us for UFME Inquiries only. For vocations and prayer requests, contact the Franciscans in your area.{% endtrans %}

In view header add "Unfiltered text" field
Click checkbox "Use replacement tokens from the first row"
Add REPLACEMENT PATTERNS in my case {{ nothing }}
Clear cache
And after that you will be able find text in user interface translation "admin/config/regional/translate"
Answer: You need to use configuration for your language. Example
Answer: You can easy translate view header and footer text using the {% trans %} Twig tag in Drupal 8
Example:
{% trans %}
  Text for translate
{% endtrans %}

And after that you will be able find text in user interface translation "admin/config/regional/translate"

Translate view header


How to lock as in, prevent change to an uploaded file

Question: I am working on a web application where file uploads and revisions are tracked.  Once a project is ready, it is submitted for an approval process.  I want to, upon "submit for approval" lock down all the projects attached files to prevent further changes.
The file uploads are handled by my own simple forms, and the files are tracked in a mySql db.
Is there anyway to set the files as read only so they can not be deleted, renamed, moved, etc. But still be viewed? (prevent change even by FTP or a cPanel file manager)
The idea is to protect the integrity of what has been approved. At the least I will be using .htaccess to prevent viewing the uploads folder directly.
Obviously, someone could SSH into the server and SUDO SU and do whatever, but, I am thinking for the less tech savy folks who need GUI.
Answer: You want application level control, not operating system level control. I.e., don't set the files to read only. Instead, have your application recognize that the project is in "submitted" status, and disable the features that would allow a user to change them.
If you're worried about somebody changing files with FTP or cPanel outside the project app then you likely have a personnel issue, not a technical one. I'm never a fan of using technology to solve a personnel issue.
That said, to additionally ensure that no OS-level changes have occurred, you can generate a hash of the contents of all the component files at the time the project is submitted, and then store those hashes in the database along with the project data. This will allow you, at any later date, to determine if the files have changed since.
Comment: It is more of an audit trail then a personnel problem.  thanks for the idea of storing the hash, that'll do!


How can I run an gls model, and perform variance modelling in R?

Question: Context:  I'm trying to analyse some data where I have four explanatory variables named "Estimulo", "Barra", "DPI" and "Viendocon" (all of them are taken as factors) and one response variable, named "Vueltasmin", which is a numeric one.
As the explanatory variables are measured repeatedly (but not in a time-dependant manner, like in a repeated measures design)  I chose a gls model. This gls strategy was selected after trying an lmer model, where I rejected Shapiro-Wilk's test for normality and had some conic-like pattern in a QQplot.
I try to modelate the variance using different strategies:  VarFixed, VarIdent,Varpower and VarExp. Now here is where I can't go on: there is some error I'm getting that I'm not understanding what does it means.
For example, I run:
m2a<-gls(Vueltasmin~DPI * Barra * Estimulo * Viendocon,weights=varFixed(~Barra),data=Datos)
and getting:
Error in glsEstimate(object, control = control) :
  computed "gls" fit is singular, rank 18.
When I try to run a VarIdent, I run:
m3<-gls(Vueltasmin~DPIBarraEstimulo*Viendocon, weights=varIdent(form=~1|Barra), data=Datos)
and get the same.
I leave the following link where you can find the script (named "Script tesis lic") and the data base I'm using (named "EXP1"):
http://dropcanvas.com/hy6ds
If anyone could help or orient me, I would be very grateful.
Thank you very much in advance,
Yair.
Comment: I'm voting to close this question as off-topic because it is about how to use R without a reproducible example.
Comment: Please add a [reproducible example](http://stackoverflow.com/q/5963269/1217536) for people to work with.
Comment: I'm sorry for not doing it at first, I have added my data base and R-script inn my question. Thanks for noticing!
Comment: Links tend to go dead & the goal of the SE system is to create a permanent repository of high quality information about (say) statistics in the form of questions & answers. Please include a small reproducible example *in* the post.
Answer: Since your data is about the repeated measurement, I suggest you to take the "nlme" or "lmer" package to analyze your data.
for example,
m2a <- lme(Vueltasmin ~ DPI * Barra * Estimulo * Viendocon, random = ~1|Subject, data=Datos, na.action = na.omit,  method = "REML")
Hope this is helpful to you!
Comment: Pcrainy Huang, first of all, thank you for your answer.

Secondly, I tried your suggestion, but got another error message:

 "Error in MEEM(object, conLin, control$niterEM) : Singularity in backsolve at level 0, block 1 "

 So I'm not sure I got it yet. I added the script and the data base I'm using in case you might find my mistake inthere. Thanks a lot again!


What is a non constructible graph?

Question: I'm working through "Groups, Graphs and Trees" by John Meier.
In Chapter 5, he states that in a Cayley Graph, $\Gamma$ (or indeed any graph), the ball of radius n centred at the vertex v, $\mathcal{B}(v,n)$, is the subgraph formed as the union of all paths in $\Gamma$ of length $\le n$ that start at the vertex $v$.
It is then stated that a Cayley graph Γ is constructible if given $n \in N$ one can construct $\mathcal{B}(e,n)$ in a finite amount of time.
My question is what is an example of a non-constructible graph?
In particular, it is implied that there are finitely generated groups which have non-constructible Cayley graphs. I cannot comprehend what such a group looks like.
Answer: To construct the Cayley graph you need to be able to solve the word problem in the group, so any group with unsolvable word problem is an example.
In fact the converse is also true - if the group has solvable word problem then you can construct the Cayley graph.
Comment: Such groups are complicated (and in some sense artificial), and constructed from Turing machines with unsolvable word problem using HNN extensions. If you really want to learn about them, I would recommend the final two chapters of Rotman's book "An Introduction to the Theory of Groups",
Comment: Would you be able to go into a specific example of such a group? I had seen the result you quoted in Meier but I am struggling to see what such groups are.
Comment: Okay, thanks. I think that intuitively had concluded these groups must be "artificial". I shall have a look at Rotman.

What is a non constructible graph?


In Drupal Commerce, how can I decrease stock of product B when buying product A with rules?

Question: I'm having some issues writing some rules for my site.
Let's say I have two products. B and C, as well as a combined product called A. Buying product A should decrease the stock of B and C by 1, but the stock of product A should also always be the lowest stock of B or C.
I can't, for the life of me, find out how to do this, and googling has turned up zero results.
Answer: Are you using the Entity Reference module?  You will somehow need to set up the relationships between A + B + C in Drupal, and Entity Reference is one Rules-module compatible way to do this.
Another more flexible but more complicated way would be to use the Relation module.


How can I prevent becoming a deadlock victim?

Question: I got this exception msg:
Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim.
The only line of my code that was implicated in the Stack Trace was the last one here:
public static DataTable ExecuteSQLReturnDataTable(string sql, CommandType cmdType, params SqlParameter[] parameters)
{
    using (DataSet ds = new DataSet())
    using (SqlConnection connStr = new SqlConnection(CPSConnStr))
    using (SqlCommand cmd = new SqlCommand(sql, connStr))
    {
        cmd.CommandType = cmdType;
        cmd.CommandTimeout = EXTENDED_TIMEOUT;
        foreach (var item in parameters)
        {
            cmd.Parameters.Add(item);
        }

        try
        {
            cmd.Connection.Open();
            new SqlDataAdapter(cmd).Fill(ds);

This is a general-purpose method that I use for all sorts of queries; I haven't changed it recently, nor have I ever seen this particular exception before.
What can I do to guard against this exception being thrown again?
Answer: You can catch the deadlock exception and retry X number of times before giving up.
There's no magic solution to avoid deadlocks. If SQL Server detects a deadlock it's going to pick one of the processes to kill. In some cases you may have had deadlocks where your process was the one that was lucky enough to continue.
You can use SQL Profiler to capture the deadlocks. I had to do this in the past to try and figure out what was actually causing the deadlocks. The less often this happens the harder it is to track down. In our test environment we just created some testing code to hammer the database from a few different machines to try and cause a deadlock.
In our case we made some changes to our indexes and modified database triggers to reduce the deadlocks as best we could. In the end we still had to implement the retries as a "just in case".
Answer: It might have helped if you had shown the SQL that was passed to ExecuteSQLReturnDataTable. Meanwhile read Minimizing Deadlocks.
Of course, you may have to also look at whatever else is contributing to the deadlock.


Hashcode comparison problem

Question: I have list of a an object which is termed as rule in our case, this object itself is a list of field for which I have to do hashcode comparison as we can't duplicate rule in the
system.
i.e Let say I have two Rules R1 and R2 with fields A & B.
Now if values of A & B in R1 are 7 and 2 respectively.
And in R2 it's 3 and 4 respectively then the process I have used to check the duplicity
of Rules in the system that is hashcode comparison fails
the method which I have used is
for(Rule rule : rules){
changeableAttrCode=0;

fieldCounter=1;

attributes = rule.getAttributes();

for(RuleField ruleField : attributes){

changeableAttrCode = changeableAttrCode + (fieldCounter * ruleField.getValue().hashCode());

fieldCounter++;

}
parameters = rule.getParameters();

for(RuleField ruleField : parameters){

changeableAttrCode = changeableAttrCode + (fieldCounter * ruleField.getValue().hashCode());

fieldCounter++;

}

changeableAttrCodes.add(changeableAttrCode);

here changeableAttrCodes where we store the hashcode of all the rules.
so can please suggest me better method so that this kind of problem does not arise in future as well as duplicity of rules in system can be seen.
Thanks in advance
Comment: Yes, and you should upvote some of the previous answers too.
Comment: correct code indentation increases code readability, easier to get help
Answer: hashcode() is not meant to be used to check for equality. return 42; is a perfectly valid implementation of hashcode(). Why don't you overwrite equals() (and hashcode() for that matter) in the rules objects and use that to check whether two rules are equal? You could still use the hashcode to check which objects you need to investigate, since two equal() objects should always have the same hashcode, but that is a performance improvement that you may or may not need, depending on your system.
Answer:
Implement hashCode and equals in class Rule.
Implementation of equals has to compare its values.

Then use a HashSet<Rule> and ask if(mySet.contains(newRule))
HashSet + equals implementation solves the problem of the non-uniqueness of the hash. It uses hash for classifying and speed but it uses equals at the end to ensure that two Rules with same hash are the same Rule or not.
More on hash: if you want to do it by hand, use the prime number sudggestion, and review the JDK code for string hashcodes. If you want to make a clean implementation try to retrieve the hashcode of the elements, make some kind of array of ints and use Arrays.hashCode(int[]) to get a hashcode for the combination of them.
Answer: Updated Your hashing algorithm is not producing a good spread of hash values - it gives the same value for (7, 2) and (3, 4):
1 * 7 + 2 * 2 = 11
1 * 3 + 2 * 4 = 11

It would also give the same value for (11, 0), (-1, 6), ... and one can trivially make up an endless number of similar equivalence classes based on your current algorithm.
Of course you can not avoid collisions - if you have enough instances, hash collision is inevitable. However, you should aim to minimize the chance for collisions. Good hashing algorithms strive to spread hash values equally over a wide range of values. A typical way to achieve this is to generate the hash value for an object containing n independent fields as an n-digit number with a base big enough to hold the different hash values for the individual fields.
In your case, instead of multiplying with fieldCounter you should multiply with a prime constant, e.g. 31 (that would be the base of your number). And add another prime constant to the result, e.g. 17. This gives you a better spread of hash values. (Of course the concrete base depends on what values can your fields take - I have no info about that.)
Also if you implement hashCode, you are strongly advised to implement equals as well - and in fact, you should use the latter to test for equality.
Here is an article about implementing hashCode.
Answer: I don't understand what you are trying to do here. With most hash function scenarios, collision is inevitable, because there are way more objects to hash than there are possible hash values (it's a pigeonhole principle).
It is generally the case that two different objects may have the same hash value. You cannot rely on hash functions alone to eliminate duplicates.
Some hash functions are better than others in minimizing collisions, but it's still an inevitability.

That said, there are some simple guidelines that usually gives a good enough hash function. Joshua Bloch gives the following in his book Effective Java 2nd Edition:

Store some constant nonzero value, say 17, in an int variable called result.
Compute an int hashcode c for each field:

If the field is a boolean, compute (f ? 1 : 0)
If the field is a byte, char, short, int, compute (int) f
If the field is a long, compute (int) (f ^ (f >>> 32))
If the field is a float, compute Float.floatToIntBits(f)
If the field is a double, compute Double.doubleToLongBits(f), then hash the resulting long as in above.
If the field is an object reference and this class's equals method compares the field by recursively invoking equals, recursively invoke hashCode on the field. If the value of the field is null, return 0.
If the field is an array, treat it as if each element is a separate field. If every element in an array field is significant, you can use one of the Arrays.hashCode methods added in release 1.5.

Combine the hashcode c into result as follows: result = 31 * result + c;
Comment: Thank you for detailed answer. Could you please tell me, which reference/guidelines you used to create this set of advices? Also I wonder, why multiply by 31? What is the magic in 31 (all bits raised)? Why multiply is better then right shift (`result = result <<< 16 + c`)?
Comment: @dma_k: I'm quoting _Effective Java 2nd Edition_, which claim that this formula is good enough in practice, without going into the mathematics. 31 is good because it's an odd prime. Also, since it's one less than a power of two, it may also be optimized to shift and subtract at the low level.
Comment: Thanks for the reference. Actually, _Effective Java_ mentions number 37, which is also a prime (primes can't be even ;). I know about the optimization when we multiply by power of 2 (can be replaced by left shift), but you are right, the answer is here: http://stackoverflow.com/questions/1074530/efficient-hashcode-implementation
Answer: I started to write that the only way you can achieve what you want is with Perfect Hashing.
But then I thought about the fact that you said you can't duplicate objects in your system.
Edit based on thought-provoking comment from helios:
Your solution depends on what you meant when you wrote that you "can't duplicate rules".
If you meant that literally you cannot, that there is guaranteed to be only one instance of a rule with a particular set of values, then your problem is trivial: you can do identity comparison, in which case you can do identity comparison using ==.
On the other hand, you meant that you shouldn't for some reason (performance), then your problem is also trivial: just do value comparisons.
Given the way you've defined your problem, under no circumstances should you be considering the use of hashcodes as a substitute for equality.  As others have noted, hashcodes by their nature yield collisions (false equality), unless you go to a Perfect Hashing solution, but why would you in this case?
Comment: I assumed he's using the hashcode the same way the `java.util.HashMap` uses it to find a Key. The hash is for finding a bucket, and then, if there are keys in that bucket uses equals for comparison. For me is completely valid creating new instances for a key that's already in a map (or another structure), and looking for that key to replace the entry or add a new one (and much more performant). The problem is he needs, aside from a good hash impl, using equals between the same-hashode ítems.
Comment: He said "I can't duplicate" in the sense "I must not", not in the sense "I'm forced to not duplicate by the running environment". So he must find a way to achieve value-level-uniqueness knowing he can fall into the not-wanted-but-physically-posible instance duplication.
Comment: @helios - first, unless you've heard something else from him than is written in his question, there's nothing to support your interpretation of the word "can't" - he literally said "we can't duplicate rule in the system".  Second, if you are right, his question is completely silly.  Why would he even think about duplicating in order to do value comparisons?  Why not just do value comparisons?  Hashcodes are absolutely not the way to go for that. But thanks for making me think about this again.
Comment: @helios - Ah, interesting.  Well, that interpretation makes more sense: hashcode as a quick shortcut.  It's still the wrong approach, until he's tried it with simple value equality and found the performance to be unacceptable, but at least it makes more sense.  Thanks.
Comment: Thanks for all the comments Perkins got it absolutly right I have to use equals method when hashcode comes out to be same.

Hashcode comparison problem


Regex para pegar texto entre <> em python?

Question: Gostaria de extrair o texto contido entre <> de uma string. De início, construí a seguinte expressão:
import re
m=re.search(r'<(.*)>','abevbv envvrhwkv <eiwbv> ebvi <wieunv> ajhbsvhj')

A expressão estaria perfeita se não houvessem dois textos entre <>. Nesse caso, terei como retorno:
'eiwbv> ebvi <wieunv'

Porém eu quero:
'eiwbv'

Que expressão regular eu teria que usar para obter esse resultado?
Answer: Coloque uma interrogação após o asterisco, assim:
m=re.search(r'<(.*?)>','abevbv envvrhwkv <eiwbv> ebvi <wieunv> ajhbsvhj')

print (m)
<_sre.SRE_Match object; span=(17, 24), match='<eiwbv>'>

DEMO
Answer: A sua RegEx está quase correta, o que deu fez dar errado é que você utilizou um quantifier greedy (guloso) no ponto (.).
Isso faz com que a regex busque até a ultima ocorrência em que pode ser encaixada, deixando sempre o grupo capturado o maior possível, você deveria ter usando um quantifier lazy (preguiçoso), fazendo com que ele sempre pare a captura na primeira ocorrencia do delimitador, no seu caso ">"
Para resolver o seu problema é necessário somente mudar o * para *?
import re
m=re.search(r'<(.*)>','abevbv envvrhwkv <eiwbv> ebvi <wieunv> ajhbsvhj')


How to remove width and height from image field images in Views?

Question: I'm using Views to generate some HTML, and this includes a field from a Views field.
My problem is, the layout needs the images to be reponsive (height and width based on the parent element), but this is being broken by Views writing in the width and height as HTML attributes which override all the CSS.

I've tried the answer to this question which uses hook_field_attach_view_alter() to remove the width and height from the output array - but it doesn't work, the width and height are still written into the HTML.
Checking the width and height values at the time hook_field_attach_view_alter() is called, it seems it's a timing problem: it looks like this is called before the image preset is applied. E.g. my image preset is 190px x 190px scale and crop: at the time hook_field_attach_view_alter() is called, the width and height are the original width and height of the image, then the 190 gets written over whatever changes the hook makes.
Is there another hook I can use for the same effect?

I also tried using Rewrite results in Views to re-create the image from its component parts. The problem is, it has access to everything except the actual image URL or URI:

Is there a way to access the image URL or URI in the replacement patterns for Views rewrite for an image field?
Comment: Have you tried using a Views template?
Comment: I thought they were for whole views, not for specific field types?
Comment: There are different levels of specificity of views templates. Click the "Theme: Information" link under the advanced settings of your view for more info. There are field-level templates for views.
Answer: It feels clumsier than I'd hoped and requires a module, but I found a solution:

Install image_url_formatter module, which creates an image field display that is just the url but still allows you to choose from available image styles
Change the image field in the view to use the new "Image URL" display.
Use the Views rewrite feature to rebuild the image ().


Send all traffic over VPN connection

Question: i want write a mac vpn client,now ,in the system network ,it has a setting named "Send all traffic over VPN connection",how to set it by code?i think it is not in SCNetworkConfiguration
Answer: here's a method i used before.  it illustrates pretty straight forwardly how to get the current ipv4 dictionary to then change it and set it back. change the CFSTR("1") to a 0 or 1 depending on your needs
<code>#define GetCasted(value, type) ((value) && (CFGetTypeID(value) == type##GetTypeID()) ? ((type##Ref)value) : NULL)

-(void)setIPv4Stuff:(SCNetworkServiceRef)service{

    SCNetworkProtocolRef protoR = SCNetworkServiceCopyProtocol(service, kSCNetworkProtocolTypeIPv4);
    CFDictionaryRef proxyDictR = SCNetworkProtocolGetConfiguration(protoR);

    const void *configMethodP = proxyDictR ? CFDictionaryGetValue(proxyDictR, kSCPropNetIPv4ConfigMethod) : NULL;

    CFStringRef configMethod = GetCasted(configMethodP, CFString);

    CFMutableDictionaryRef newProxyDictR;

    newProxyDictR = CFDictionaryCreateMutable(NULL, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);

    CFDictionarySetValue(newProxyDictR, kSCPropNetIPv4ConfigMethod, configMethod);
    CFDictionarySetValue(newProxyDictR, kSCPropNetOverridePrimary, CFSTR("1"));

    SCNetworkProtocolSetConfiguration(protoR, newProxyDictR);

    CFRelease(newProxyDictR);

}
</code>
Answer: kSCPropNetOverridePrimary to 0 should disable this (you can look up that key in the ipv4 dictionary).
You need to set protocol configuration for the ipv4 protocol with that key set to 0 or 1 to disable or enable that flag respectively.

Send all traffic over VPN connection


How can I get the form group validation result in the template driven form?

Question: I refer the following web page for create a form group in a template-driven form:
https://www.tektutorialshub.com/angular/formgroup-in-angular/
I have a template-driven form.
I embed a form group in this form.
How can I get the form group validation result?
Here is my code.
Why the following code always returns true? even I empty all input form fields.
console.log(this.manualGroup.errors===null);

Answer:
Why the following code always returns true?

manualGroup is not able to have its own validation because it is not a FormControl. And FormGroup doesn't have errors of its children either. I guess you were expecting that.
If you want just to get validation of FormGroup, use this.manualGroup.valid instead.


Cartesian product of Hamiltonian graphs

Question: Prove that cartesian product of 2 Hamiltonian graphs is also Hamiltonian.
Also please explain the significance of cartesian product of 2 graphs
Comment: What kind of product?  $\times$, $\square$, or $\boxtimes$?  What do you mean by significance?  That's an awful broad term...
Comment: What do you understand about the cartesian product of graphs?  I suspect this question is mostly to gauge what you understand about graph products... once you understand a little about what the product is, it will be easy to see that the statement is true.
Comment: By product of two graphs G1,G2 , I mean G1 X G2
Comment: By product of two graphs G1,G2 , I mean G1 X G2 i.e Suppose G1 (V1,E1) and G2(V2,E2), then G1 X G2 has the vertex set V3=(u1,u2), u1∈ V1 and u2 ∈ V2 and 2 vertices (u1,u2) ,(u3,u4) are adjacent iff u1==u3 and u2 and u4 are adjacent in G2 or if u2==u4 and u1 and u3 are adjacent in G1
Comment: What you have just described is the $\square$ product of graphs... not the $\times$ product.
Comment: Oh sorry, please answer for G1  ◻ G2
Answer: Here's a pair of big hints which will get you started in the right direction.
First suppose that $G_1$ has $m$ vertices and $G_2$ has $n$ vertices.  You can view $G_1\square G_2$ as an $m\times n$ grid of vertices.  In each column is a copy of $G_1$ in each row is a copy of $G_2$.  It will help your understanding if you draw a few examples, draw the product of short two short paths, maybe the product of two small cycles, maybe a path and a cycle.  That'll give you decent enough intuition to answer the question.
Second, if $G_1$ has $m$ vertices and is Hamiltonian, then $C_m$ (the cycle on $m$ vertices) is a subgraph of $G_1$.  Likewise, if $G_2$ has $n$ vertices and is Hamiltonian then $C_n$ is a subgraph of $G_2$.  This implies that $C_m\square C_n$ is a subgraph of $G_1\square G_2$.  Then, all you need to show is that $C_m\square C_n$ is Hamiltonian and you are done (because you find the Hamiltonian cycle in a subgraph of $G_1\square G_2$).  Your intuition from step 1 will help a lot with step 2.
Comment: thanks a lot for helping me out

Cartesian product of Hamiltonian graphs


Java Programming: Turtle not moving in desired direction

Question: I'm writing a Java program in which I have to get the turtle to move West, but what I have so far isn't getting the turtle to move in the desired direction.
public static void moveTurtleWest(Turtle t, int n)
    {
        for(; n > t.getX();n-- ){
            t.moveWest();
            }
    }

Here is the rest of my code:
package assignment;

import java.text.MessageFormat;
import java.util.Scanner;

import java.awt.Point;
import java.awt.Dimension;
import java.awt.Rectangle;

import java.time.LocalDate;

/**
 * This class represents a turtle, like a Logo-like turtle, which keeps its position.
 * The turtle keeps its x and y position, and has methods moveNorth() etc which change it
 * Assume normal math coordinates and map convention, so as you move up (North) y increases, and as you move East x increases.
 *
 */
class Turtle {

    // HINT - you may want to have variables to keep the position. Keep these variables private,
    private int x,y;

    // TODO - The empty constructor initializes position to 10,10
    public Turtle() {
        this.x = 10;
        this.y = 10;
    }

    public int getX() {
        // TODO - implement this
        return this.x;
    }
    public int getY() {
        // TODO - implement this
        return this.y;
    }

    public void moveNorth() {
        // TODO - implement this. this increments the y coordinate by one unit
        this.y += 1;
    }

    public void moveSouth() {
        // TODO - implement this. this decrements the y coordinate by one unit
        this.y -=1;
    }

    public void moveEast() {
        // TODO - this increments the x coordinate by one unit
        this.x +=1;
    }

    public void moveWest() {
        // TODO - this decrements the x coordinate by one unit
        this.x -=1;
    }

    public String toString() {
        return "Turtle[x="+getX()+", y="+getY()+"]";
    }

    public boolean equals(Turtle t)
    {
        // TODO - you need to implement this
        // two turtles are equal if their X and Y coordinates are equal.
        if (t == null) {
            return false;
        }
        if (!(t instanceof Turtle)) {
            return false;
        }
        return (x == ((Turtle) t).x && y == ((Turtle) t).y);
    }

    public void move(String direction)
    {
        // TODO - you need to implement this
        // move to the right direction; direction can be North, South, East, West
        moveEast();
    }
}

public class Assignment7 {
    // TODO - you need to implement this. Move the given turtle to the West, n times
    public static void moveTurtleWest(Turtle t, int n)
    {
        for(; n > t.getY();n-- ){
            t.moveWest();
            }
    }

    // TODO - you need to implement this. Move the given turtle to the East, n times
    public static void moveTurtleEast(Turtle t, int n)
    {
    }

    // TODO - you need to implement this. Move the given turtle to the North, n times
    public static void moveTurtleNorth(Turtle t, int n)
    {
    }

    // TODO - you need to implement this. Move the given turtle to the South, n times
    public static void moveTurtleSouth(Turtle t, int n)
    {
    }

    // TODO - you need to implement this. Move the turtle to the asked position, by calling MoveXXX etc
    public static void moveTurtleTo(Turtle t, int x, int y)
    {

    }
    public static void main(String[] args) {
        // you can use this as you wish to test or exercise your function. Not graded.
        Turtle t=new Turtle();
        moveTurtleTo(t,15,16);
        System.out.println(t);
    }
}

Here is the code that is executing when I run this method in git Bash:
public void testMoveWest()
    {
        Turtle t=new Turtle();
        t.moveWest();
        t.moveWest();
        t.moveWest();
        t.moveWest();

        Assert.assertEquals(6,  t.getX());
    }

    @Grade(points=10)
    @Test

Any help is appreciated and points will be rewarded.
Thanks in advance!
Comment: Is it the opposite direction than what you expect? Is this done via GUI (like Swing) interface?
Comment: so the test that you have mentioned that itself if not passing?
Comment: can you print the value of getX() in that test? or you can find the same in the output of assertEquals fail.
Comment: No, I'm running a script in GitBash in which if my code executes, then I will be awarded 10 points, so I'm running my validation and compiler via command line.
Comment: Exactly the test is not passing
Comment: The test is not passing because the for loop I wrote up top is incorrect because the public void testMoveWest is not executing.
Comment: No I cannot print the value of getX() in that test. My code compiles it is just not executing the  public void testMoveWest correctly because I'm not getting the 10 points.
Comment: so your actual problem is that for loop is incorrect. I  dont see anything wrong in your moveWest method.
Comment: Yes but I need something in the
public static void moveTurtleWest(Turtle t, int n) method in order for my program to work correctly.
I need a for loop or something to get this method working please
Comment: try the loop which I have mentioned in answer and let me know whether your test passes properly or not.
Comment: It passed!
Thanks again
Answer: Possibly correct for loop:
 public static void moveTurtleWest(Turtle t, int n) {
         for(int i=0; i <n;i++ ){
               t.moveWest();
          }
 }

Comment: Great Job!. I greatly appreciate all of the help!
Answer: Shouldn't the predicate in the for loop be t.getX() instead of t.getY()?
Comment: Yes it should be and I made the change but it is still not working


LazyInitializationException with map collection

Question: I know that it is a trite question, but I could not find a solution. I have two beans and one of them has HashMap collection. I'm getting an exception when trying to read this collection. Mapping config had been specified to load this collection eagerly.
My environment is :

Hibernate 4.2.0
mysql-connector-java 5.1.24

Also I have two beans:
public class FeaturedDoc {
    private Long id;
    private Map<Feature, Float> features;

    public FeaturedDoc() {
        features = new HashMap<Feature, Float>();
    }
   (getters and setters)
}

and
public class Feature {
    private Long id;
    private String name;
    private Long internalId;
    (getters and setters)
}

This beans have mapping:
<class name="Feature" table="FEATURE">
    <id name="id" type="long" column="ID">
        <generator class="increment"/>
    </id>
    <property name="name" length="255" type="string" unique="true" column="NAME" index="INDEX_NAME"/>
    <property name="internalId" type="long" unique="true" not-null="false" column="INTERNAL_ID" index="INDEX_INTID"/>
    <sql-insert>insert into FEATURE (NAME, INTERNAL_ID, ID) values (?, ?, ?) on duplicate key update ID = ID</sql-insert>
</class>

<class name="FeaturedDoc" table="FEATURED_DOC">
    <id name="id" type="long" column="ID">
        <generator class="increment"/>
    </id>
    <map  name="features" table="DOC_FEATURE" cascade="all" lazy="false" fetch="join">
        <key column="ID"></key>
        <map-key-many-to-many column="FEATURE_ID" class="Feature"/>
        <element column="value" type="float"/>
    </map>
</class>

Also I have DAO layer with method:
public FeaturedDoc read(long id) {
    FeaturedDoc fd = null;
    try {
        session.beginTransaction();
        fd = session.get(FeaturedDoc.class, id);
        session.getTransaction().commit();
    } catch (Exception e) {
        e.printStackTrace();
        session.getTransaction().rollback();
    } finally {
        close();
    }
    return fd;
}

When I'm trying to do something like this:
FeaturedDoc fd = daoService.read(26);
System.out.println(fd.getFeatures());

I'm getting an exception

org.hibernate.LazyInitializationException: could not initialize proxy - no Session

Do you know how should I fix this error?
Comment: Do you know how should I fix this error?
Answer: Assuming that everything else is okay (in the mappings), have you tried putting lazy="false" before cascade="all". I found that this was a problem in my mapping which resulted in this LazyInitializationException error.
This ordering is shown in the following Hibernate Reference: http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/collections.html
Comment: Yes, I've tried, but it doesn't work. Actually I don't think that the order of attributes is the matter.
Answer: I have solved this problem! The reason was in Feature class. It had not hashCode and equals functions. After implementation of these functions everything has become ok.


Apply a Moving average to different closing prices of currencies simultaneously

Question: I want to apply a single moving average to different closing prices at the same time on my Data Frame.
Open    Close   ... High    Low
AUDUSD=X    EURUSD=X    GBPUSD=X    NZDUSD=X    USDCAD=X    USDCHF=X    USDJPY=X    AUDUSD=X    EURUSD=X    GBPUSD=X    ... USDCAD=X    USDCHF=X    USDJPY=X    AUDUSD=X    EURUSD=X    GBPUSD=X    NZDUSD=X    USDCAD=X    USDCHF=X    USDJPY=X
Date
2019-01-02  0.704722    1.146132    1.275234    0.671362    1.36250 0.98180 109.660004  0.704791    1.146171    1.275429    ... 1.36596 0.98925 109.674004  0.698470    1.134572    1.258463    0.666298    1.35690 0.9794  108.730003
2019-01-03  0.692008    1.131734    1.251878    0.663398    1.36300 0.98916 107.449997  0.691898    1.131811    1.252191    ... 1.36440 0.98977 108.271004  0.691946    1.131734    1.251878    0.661559    1.34736 0.9845  106.776001
2019-01-04  0.700722    1.139095    1.262897    0.668852    1.34760 0.98726 107.865997  0.700624    1.139108    1.262881    ... 1.34906 0.99068 108.583000  0.699496    1.134816    1.261782    0.667361    1.33814 0.9852  107.513000
2019-01-07  0.712606    1.141292    1.273285    0.674450    1.33689 0.98577 108.488998  0.712378    1.141044    1.273496    ... 1.33760 0.98577 108.625000  0.711551    1.140524    1.272426    0.673179    1.32882 0.9789  108.036003
2019-01-08  0.714602    1.147934    1.278691    0.675548    1.32967 0.97929 108.642998  0.714592    1.147974    1.278609    ... 1.33190 0.98333 109.077003  0.711789    1.142465    1.271488    0.670979    1.32670 0.9788  108.449997

forex_data_1['Moving average'] = forex_data_1['Close'].rolling(window=30).mean()

ValueError: Cannot set a DataFrame with multiple columns to the single column Moving average
Comment: Please format your question properly. It is compleately unreadable like this.
Answer: Your attributes concering the df (rolling, mean) generally works.
The problem is your df itself, it seems that you don´t adress it correct.
I guess, forex_data_1['Close'] is no column. For deeper analysis, you must post your dataframe in a better readable way. As mentioned by @atteggiani, without this, analysis for your error is not possible.
This works fine for me:
import pandas as pd
df=pd.DataFrame({'A':[1.2,1.3,1.4,1.5,7],'B':[2,3,4,5,6]})

df['rollmean'] = df['A'].rolling(window=2).mean()

Result:
     A  B  rollmean
0  1.2  2       NaN
1  1.3  3      1.25
2  1.4  4      1.35
3  1.5  5      1.45
4  7.0  6      4.25

Solution with iteration over rows as asked:
for c in df.columns:
    df['rm'+c]=df[c].rolling(window=2).mean()

Solution after specific definition of df from questioner forex_data.head():
for c in forex_data.columns:
    forex_data['rm'+c[0]+c[1]]=forex_data[c[0]][c[1]].rolling(window=2).mean()

... fills separate columns at the end of the table with prefix rm and addresses each tuple of multicolumns. The columns in your dataframe are multicolumns. You adddressed in your example only one attribute of the column, not both. If you insert print(c) directly under the for-loop you understand what I mean.
Comment: moved to my answer ...
Comment: yours is nice and simple. can you apply your rollmean to both 'A' and 'B' at the same time?
Comment: I see your response and is helpful.


What is harmonic mixing / mix in key?

Question: I'm currently learning how to become a DJ. One of the concepts that caught my attention was "harmonic mixing". I know some very basic music theory, and I'm having some trouble understanding it.
What is harmonic mixing? How can I use it on my DJ sets? Do I need to know music theory to understand and use it?
Answer: Harmonic mixing is the practice of using music theory in your dj sets.
You can use this knowledge to achieve specific functions when mixing two songs (similar to chord progressions), or to know which songs are compatible with each other, just to give a few examples.
The most common and basic form of harmonic mixing.
If you don't want to know about the ins and outs, just skip to "The basic and common form of harmonic mixing for dummies".
This is its most common and basic form, which tends to be the first thing to be suggested:

Song X is compatible with song Y if:

Both songs share key signatures.
Y's key is a fifth above, and both have same quality.
Y's key is a fourth above, and both have same quality.

Sharing quality means that both songs are in minor, or both in major. There are many other qualities, but those two are by far the most common.
This means that a song in A minor will be compatible with:

Another song in A minor, or a song in C major.
Another song in E minor.
Another song in D minor.

In short, you can mix a song in A minor with songs in: A minor, C major, E minor, or D minor. Don't forget that this is only the most common and basic form of harmonic mixing. Song interactions are much more complex than that, but this is a good rule of thumb that is a common place to start in.
Why those suggestions?
If you are not interested in the theory behind the suggestions, skip to "The basic and common form of harmonic mixing for dummies".
Short answer: The songs will be using either the same notes or will be using only one different note.
Long answer:
(In order to make this as simple as possible, I'll base all examples in either C major or A minor.) The explanation for each suggestion is:

Both songs use the same notes. Both songs share key signature. C major is the relative major of A minor, and A minor is the relative minor of C major.

A minor = A B C D E F G
C major = C D E F G A B

Both songs share almost all notes but one, and that one note that is not shared is compatible with each other's mode (it sounds good).

A minor = A B C D E F G
E minor = E F# G A B C D

The difference is: F# or F.
F# is the major sixth of A. A minor with a major sixth instead of a minor sixth is the A dorian mode. It's a note that is very compatible with A minor.
F is the minor second of E. E minor with a minor second instead of major second is the E phrygian mode. Again, it's a note that is very compatible with E minor.

Both songs share almost all notes but one, and that one note that is not shared is compatible with each other's mode. (yes, same as suggestion 2).

A minor = A B C D E F G
D minor = D E F G A Bb C

The difference is: Bb or B.
Bb is the minor second of A. A minor with a minor second instead of major second is the A phrygian mode. It's a note that is very compatible with A minor.
B is the major sixth of D. D minor with a major sixth instead of minor sixth is the D dorian mode. Again, it's a note that is very compatible with D minor.

For completion, let's see a similar example but with a major quality.
According to our 3 suggestions, a song in C major will be compatible with:

Another song in C major or a song in A minor.
Song in G major.
Song in F major.

The same explanations apply. Let's see a quick review:

Songs share same notes (etc).

C major = C D E F G A B
A minor = A B C D E F G

Songs share almost all notes but one (etc).

C major = C D E F G A B  (difference is F# and C D E F# G A B = C lydian)
G major = G A B C D E F# (difference is F and G A B C D E F = G mixolydian)

Songs share almost all notes but one (etc).

C major = C D E F G A B (difference is Bb and C D E F G A Bb = C mixolydian)
F major = F G A Bb C D E (difference is B and F G A B C D E = F lydian)

The basic and common form of harmonic mixing for dummies: using a circle of fifths diagram.
There is another approach to basic harmonic mixing. It has the advantage of being very simple, so it can be used by almost anyone without much music theory knowledge (if any). It has the strong disadvantage of limiting your knowledge to following a diagram. It won't get you further. It will make your set work musically, but you won't understand why, and you won't be able to advance in either harmonic mixing or music theory. I don't recommend this method, but I include it anyway for completion and because I know there are many DJs out there that don't have much interest in music theory.
It has only one simple suggestion. Follow this diagram:

The diagram works as follows:
Locate the key of the song in the diagram. That song is compatible with:

Songs that have the same key.
Keys that are to the side of your song's key.
The key exactly bellow or above the key of your song.

In the case of C major the keys to the side are F major and G major. The key exactly bellow it is A minor. So, in short, your song is compatible with the keys in its immediate vicinity, the ones it shares a border with:

Most common and basic form of harmonic mixing, part 2 (includes part 2 for dummies)
There is an extra suggestion that is commonly made once part 1 is understood:

A song is also compatible with the 4th and 5th grade of the relative major/minor.

Explanation:

C major = C D E F G A B
A minor (relative minor of C major) = A B C D E F G
D minor (4th grade of A minor) = D E F G A Bb C
E minor (5th grade of A minor) = E F# G A B C D

Again, we see that C major shares almost all notes but one with D minor and E minor. The difference notes are Bb and F#. C D E F G A Bb = C mixolydian. C D E F# G A B = C lydian. The difference notes, again, sound kinda cool. They are compatible.
For dummies:
This adds two other compatible notes in the circle of fifths diagram too. Now the song is also compatible with the keys in its vicinity including the ones in the corners. C major is compatible with F major, G major, A minor, D minor, and E minor.
What this means for the circle of fifths diagram system is that two songs are compatible if they are neighbors in the circle of fifths diagram:

Beyond the most simple form of harmonic mixing.
We have seen a very basic form of musical interaction between two songs. It works, but if you want to do more interesting and less restrictive things, you have to go beyond it.
What is beyond? It is simple: knowledge of music theory and its applications to your instrument. In this case the turntables, the CD players, Ableton Live, Traktor, Serato, whatever you are using. Once you understand basic music theory you can apply it as you would with any other instrument. You can make progressions, you can make chords, you can make scales.
Let's see some examples.
We have song X in B minor and song Y in C minor. According to our basic suggestions both songs are not compatible, but music in real life is not that simple. Let's assume we are mixing the outro of song X with the intro of song Y. What is being played at that time, on each song? Maybe there are no tonal elements at all! Maybe there is only one tonal element on each song. If that's the case, maybe the only note that is being played in song X is B, and the only note that is being played in song Y is C. You can mix those portions of the songs and it will work. It's just a C and a B. If there is only one tonal element on each song, it will work. Which pair of notes do not sound good? It's when more than two notes interact when things get more complex, but even there things might or might not work depending on the specific notes that are being played on the moment of the mix, and not the overall key of the songs.
In fact, you might want to go beyond writing down the key of the overall song and start documenting the notes that are being played in the intro, mid, and outro of the song (or whatever division you can think of). That way you will not only know the keys of the songs, but the actual notes that are being played on a specific part. You can make chords and scales with this information.
This way you think about the overall keys of the songs as chord progressions, and the specific tonal dynamics of parts of the songs that are being mixed as tonal qualities.
Song X and Y don't have tonal elements in the parts you are mixing them, but they have tonal elements in other parts? Then think about them as chord progressions. They will sound well together, but will they work when the tonal elements of Y start? How will the change feel? Maybe you only had a few moments without tonal elements, and when the new music kicks in it won't work very well with the previous context. Maybe there was a long time without tonal elements and the context was long gone, so you can introduce a new harmony without much worry.
Another example: Song X is in C major and song Y is in B major. According to the suggestions we have seen, they are not compatible. We will mix the outro of X with the intro of Y, and the notes being played at the outro of X are C and E, and the notes being played at the intro of Y are B and E. C, E, B. That's Cmaj7 without the 5th! It, after all, might work!
Going beyond the most simple form of harmonic mixing is understanding harmony and music theory.
Comment: Wow. I had no idea this was such a big deal. Here in the classical world, we just call the "Camelot Wheel" a circle of 5ths. There seems to be a lot of marketing around the Camelot Wheel specifically, but it all looks good on paper and seems to be predominant in the discipline. +1 for a veritable encyclopedia article of an answer.
Comment: Well, [Camelot Sound](http://www.camelotsound.com) seems to have been the original developer, hence the name, and [MixedInKey](http://www.mixedinkey.com) appears to be a primary vendor. "Harmonic Wheel" would be a perfectly appropriate generic name, but it's not really a bad thing that DJ's know it only as the Camelot Wheel -- they're just marketing software, not the theory itself.
Comment: @NReilingh Yes, the "Camelot Wheel" is just two circle of fifths, aligned in relation with the relative major/minor. I try to avoid the "Two circle of fifths aligned in relation of the relative major/minor" or even "Two circles of fifths" name because it tends to scare Djs that just started diving into harmonic mixing. That's why I use "Camelot Wheel" instead. Didn't know it was being marketed, though. Maybe it's time to find a new name for the Camelot Wheel?
Comment: @NReilingh Yeah that always bothered me. I'm changing the answer to use a generic circle of fifths instead of the camelot system.

What is harmonic mixing / mix in key?


Atualizar navegador VBA

Question: Tenho umamacro q faz webscrapp no internet Explorer, mas o site é bem ruim e às vezes trava. Gostaria de colocar no loop um código que força o IE a atualizar se demorar muito pra carregar, travar ou não der um erro que ative o resume next.
o loop eh o seguinte:
Preenche o nome do cliente > clica em pesquisar. Nesse momento a trava (quando acontece) acontece de tres formas: Às vezes o IE fica rodando, as vezes ele retorna servidor não encontrado, e as vezes retorna cliente não localizado, e nesses casos a Macro não considera erro, e não segue pra próxima linha.
Eu gostaria q iniciasse um timer logo após clicar em pesquisar, pra atualizar o navegador se demorar tipo 2 minutos pra reiniciar o loop. A rotina on error não processa quando o IE retorna servidor não encontrado, e a Macro só volta a rodar se eu atualizar manualmente o navegador.
Answer: Se o site é bem ruim então você está em maus lençóis, pois, o IE também o é...
Quando utilizava o IE para webscraping fiz a função abaixo. Ainda assim há o risco de se enfrentar um loop eterno. Hoje estou aprendendo a lidar com o Selenium, manipulando o Chrome Driver.
<code>Function fnCheckDocReadyState(ByRef objIE As InternetExplorer) As Boolean
    Dim curTimer As Currency
    Dim oDoc As MSHTML.HTMLDocument

    curTimer = Timer
    On Error Resume Next
    Set oDoc = objIE.document
    On Error GoTo 0
    If Not (oDoc Is Nothing) Then
        Do While objIE.document.readyState <> "complete"
            DoEvents: DoEvents: DoEvents
            If Timer - curTimer > 10 Then
                objIE.Refresh2
                curTimer = Timer
            End If
        Loop
        fnCheckDocReadyState = True
    End If
lblSair:
    Exit Function
End Function
</code>

Atualizar navegador VBA


Digital effects processors with VST plugins?

Question: I can't find how and if people process VST effects in real time. Is there such a thing where you can load any chain of VST effects? It doesn't make sense to me how such a thing couldn't exist.
Comment: Related question (despite title, not Reaper specific): [How to get rid of latency experienced while using VST in Reaper?](https://music.stackexchange.com/q/48574/70803).
Answer: Even running in a computer, VST effects are usually capable of running in real time, if by real time you mean 'live' rather than 'offline'. However it is true that most computer platforms introduce a comparatively large amount of latency, which may be unacceptable in some cases.

Is there such a thing where you can load any chain of VST effects?

If you mean a physical device that allows you to load VST effects and typically run them with less latency than a general-purpose computer - yes, such things do exist. Examples are the Muse Receptor and Seelake Audiostation.
Note that even these - and all digital effects - will introduce some latency when placed in an analogue signal chain, due to the A/D and D/A processes. So depending on how much latency is acceptable to you, you might consider that no digital effects are truly 'real-time' in the sense of being latency-free.


What is the result of averaging the weights of Linear Models?

Question: Say I have two datasets that have the same features but different samples. If I build two linear models, one for each of the datasets, and then take a weighted average (say the weights here are the number of samples in each of the datasets that the model was built on) of the weights of each of these linear models, what can I say about the resulting meta-model?
Model 1, built over the first $m$ samples
$$
Y_0 = \sum_{i=0}^{k}(\beta_{0,i}*x_{i})
$$
Model 2, built over the remaining samples ($n$ in this case)
$$
Y_1 = \sum_{i=0}^{k}(\beta_{1,i}*x_{i})
$$
The merged model that I am interested in
$$
Y_{1+0}= \sum_{i=0}^{k}(\frac{m*\beta_{0,i}+n*\beta_{1,i}}{m+n}*x_{i})
$$
I have looked into ensembling, and as far as I can tell it is common practice to use multiple models for prediction (average the result) or classification (majority vote), but I have yet to find someone discussing what the result of a merged model would be.
Comment: Is your question about *this particular process of weighting* the models (there are better ones that have theoretical justification, btw) or is it really concerned about *whether and how* to combine regression results?
Comment: As a minor correction, I think you need to redo your final equation to fix the denominator, e.g. divide by $m+n$ instead of $2$, to get the weighted average.
Comment: Note that for linear models averaging the coefficients in this way is equivalent to averaging the predictions
Comment: The right way to average weights the predictions by the inverses of their variances.  Those usually will not be the same as the data-count weights. One way to see why not is to consider the case of simple regression where the ranges of the $x_i$ don't even overlap between the two fits.  It should be clear that the greatest weight should be given to the coefficient from the applicable fit.  The result won't even be a linear function of $x$!
Comment: @whuber it is a bit of both I suppose. I would be interested in the better model merging procedures, but I would also like to understand what the follies of this procedure are.
Answer: Its not clear why you'd want to train two separate models on two samples coming from the same population. If your goal is to check for consistency of results from both models, you might want to carefully choose an appropriate sampling method so that m and n are not too heterogeneous. If they are heterogenous, in other words the underlying data distribution is different; that would result in different parameter estimates and combining the models won't be such a good idea.
Also the above method would be tricky to implement when you have longitudinal data or time series data.
Comment: Sampling of Time Series data is tricky because you can't just partition the data on time dimension. For example, if there are inherent seasonal patterns in the data, sampling may lead to truncated time-series with or without those effects.
Comment: I don't, Nate, but the formulas have appeared here in several places.  Naturally, it's easiest for me to remember and find one of my own contributions at https://stats.stackexchange.com/questions/51622/combining-two-covariance-matrices/51927#51927, but I know there are some other posts on the subject, too.  Here's one search for them: https://stats.stackexchange.com/search?q=covariance+combine.
Comment: My use case would be very large datasets where it is expensive to move the data from one data center to another, where as sending the weights of a model built over each of the respective datasets would be cheap. I am assuming the underlying distributions are homogeneous. Why do you say it would be difficult with longitudinal and time series data?
Comment: How many variables appear in the model?  Unless it's huge, it ought to be practicable to communicate the SSP matrices, which is all you need to update the estimates.  With $p$ variables these have $(p+1)(p+2)/2$ coefficients.
Comment: @whuber, do you have any literature suggestions describing the process for computing that update?


ng-repeat within Grid-A-Licious Grid

Question: I am using the Grid-A-Licious plugin (http://suprb.com/apps/gridalicious/).
My markup is as follows:

Properties Around you

{{m.title}}

                            Bath : 2
                            Beds : 2
                            Pool : 2
                            $380,000

                            See Detail
                            Compare

As you can see inside the grid I have a ng-repeat property which is bound to my view model.
The issue I am encountering, is that Grid-A-Licious requires the following code to be ran on document.ready():
$(".grid-item").gridalicious({
        width: 250,
        gutter: 10,
        animate: true,
        effect: 'fadeInOnAppear'
    });

    $(".grid-galeries").gridalicious({
        width: 240,
        gutter: 10,
        animate: true,
        effect: 'fadeInOnAppear'
    });

The problem I am having is that the content of my grid is bound to my view model, and will change dynamically depending on other conditions (it is actually the result of a service call).
This results in the grids appearance being wrong.
Is there any way, I can call the above selectors, each time the model property the grid elements are bound to changes?
And does doing this totally go against angularJS conventions?
Many Thanks in advance
Answer: You can use watch to achieve your goal. However, please note that this should be handled in a Directive  and not in a controller but to make this a short(more direct) answer I will use a controller sample:
app.controller('myCtrl',['$scope',function($scope){
    "use strict";

    $scope.map.dynamicMarkers= [{ /*some object we do not know*/ }];

   $scope.$watch("map.dynamicMarkers",function(newValue, oldValue, scope){
       $(".grid-item").gridalicious({
         width: 250,
         gutter: 10,
         animate: true,
         effect: 'fadeInOnAppear'
       });

      $(".grid-galeries").gridalicious({
         width: 240,
         gutter: 10,
         animate: true,
         effect: 'fadeInOnAppear'
       });
   },true);
}]);

Disclaimer: The modification of the view from your controller is not recommended nor it is the "angular way". I think these actions should be moved into a directive.

ng-repeat within Grid-A-Licious Grid