LF-RCNN, MN-RCNN, MN-RCNN-Wt
EXTRACTIVE
LF-RCNN, MN-RCNN, MN-RCNN-Wt
EXTRACTIVE
We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0. Models ::: LF-RCNN Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture. Models ::: MN-RCNN Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6) Models ::: MN-RCNN-Wt Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
For image features, we extract Faster R-CNN features with ResNet-101 backbone trained on Visual genome BIBREF4 dataset, similar to BIBREF2. We use an adaptive number of object proposals per-image ranging from 10 to 100 generated using a fixed confidence threshold and each object is then associated with 2048-dimensional mean-pooled features using ROI pooling. We use discriminative decoding throughout our models. We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0. Models ::: LF-RCNN Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture. Models ::: MN-RCNN Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6) Models ::: MN-RCNN-Wt Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
2001.05865
Introduction </s> Visual Ġdialog ĠB IB REF 0 Ġis Ġan Ġinteresting Ġnew Ġtask Ġcombining Ġthe Ġresearch Ġefforts Ġfrom ĠComputer ĠVision , ĠNatural ĠLanguage ĠProcessing Ġand ĠInformation ĠRet ri eval . ĠWhile ĠB IB REF 1 Ġpresents Ġsome Ġtips Ġand Ġtricks Ġfor ĠV Q A Ġ2 . 0 ĠChallenge , Ġwe Ġfollow Ġtheir Ġguidelines Ġfor Ġthe ĠVisual ĠDial og Ġchallenge Ġ2018 . ĠOur Ġmodels Ġuse Ġattention Ġsimilar Ġto ĠB IB REF 2 Ġto Ġget Ġobject Ġlevel Ġimage Ġrepresentations Ġfrom ĠFaster ĠR - CNN Ġmodel ĠB IB REF 3 . ĠWe Ġexperiment Ġwith Ġdifferent Ġenc oder Ġmechanisms Ġto Ġget Ġrepresentations Ġof Ġconvers ational Ġhistory . </s> Mod els </s> Common Ġto Ġall Ġthe Ġmodels , Ġwe Ġinitialize Ġour Ġembed ding Ġmatrix Ġwith Ġpre - trained ĠGl ove Ġword Ġvectors Ġof Ġ300 Ġdimensions Ġusing Ġ6 B Ġtokens Ġ. ĠOut Ġof Ġ113 19 Ġtokens Ġpresent Ġin Ġthe Ġdataset , Ġwe Ġfound Ġ188 Ġtokens Ġmissing Ġfrom Ġthe Ġpre - trained ĠGl ove Ġembed d ings , Ġso Ġwe Ġmanually Ġmap Ġthese Ġtokens Ġto Ġwords Ġconve ying Ġsem antically Ġsimilar Ġmeaning , Ġe . g . Ġwe Ġmap Ġover Ġten Ġvariations Ġof Ġthe Ġword ĠâĢ ľ yes âĢ Ŀ Ġ- Ġmiss pelled Ġor Ġnot Ġpicked Ġup Ġby Ġtoken izer Ġ- ĠâĢ ľ * yes ", ĠâĢ ľ yes a ", ĠâĢ ľ y ess ", ĠâĢ ľ y tes ", ĠâĢ ľ yes - ", ĠâĢ ľ yes 3 ", ĠâĢ ľ y yes ", ĠâĢ ľ y ees ", Ġetc . </s> For Ġimage Ġfeatures , Ġwe Ġextract ĠFaster ĠR - CNN Ġfeatures Ġwith ĠRes Net - 101 Ġbackbone Ġtrained Ġon ĠVisual Ġgenome ĠB IB REF 4 Ġdataset , Ġsimilar Ġto ĠB IB REF 2 . ĠWe Ġuse Ġan Ġadaptive Ġnumber Ġof Ġobject Ġproposals Ġper - image Ġranging Ġfrom Ġ10 Ġto Ġ100 Ġgenerated Ġusing Ġa Ġfixed Ġconfidence Ġthreshold Ġand Ġeach Ġobject Ġis Ġthen Ġassociated Ġwith Ġ2048 - dimensional Ġmean - pool ed Ġfeatures Ġusing ĠRO I Ġpool ing . ĠWe Ġuse Ġdiscrim in ative Ġdecoding Ġthroughout Ġour Ġmodels . </s> We Ġfirst Ġdescribe Ġour Ġmodels Ġindividually Ġand Ġthen Ġthe Ġen semb ling Ġtechnique Ġthat Ġwe Ġemploy . ĠIn Ġthe Ġfollowing , ĠMN Ġdenotes ĠMemory ĠNetworks Ġto Ġencode Ġconvers ational Ġhistory , ĠRC NN Ġsignify ĠR - CNN Ġfor Ġobject Ġlevel Ġrepresentations Ġof Ġan Ġimage , ĠW t Ġrepresents Ġadditional Ġlinear Ġlayer Ġin Ġthe Ġdec oder , Ġand ĠLF Ġa Ġlate Ġfusion Ġmechanism Ġas Ġdefined Ġin ĠB IB REF 0 . </s> Mod els Ġ: :: ĠLF - RC NN </s> Late Ġfusion Ġenc oder ĠB IB REF 0 Ġwith Ġconc aten ated Ġhistory . ĠWe Ġuse Ġtwo - lay ered ĠL ST Ms Ġwith Ġ512 Ġhidden Ġunits Ġfor Ġembed ding Ġquestions Ġand Ġhistory . ĠThe Ġobject - level Ġfeatures Ġare Ġweighed Ġusing Ġonly Ġquestion Ġembed d ings . ĠThe Ġword Ġembed d ings Ġfrom ĠGl ove Ġvectors Ġare Ġfrozen Ġand Ġare Ġnot Ġfine - tun ed . ĠFigure ĠFIG REF 6 Ġgives Ġan Ġoverview Ġof Ġthe Ġarchitecture . </s> Mod els Ġ: :: ĠMN - RC NN </s> Memory Ġnetwork Ġenc oder ĠB IB REF 0 Ġwith Ġbi - direction al ĠGR Us Ġand Ġword Ġembed d ings Ġfine - tun ed . ĠObject - level Ġfeatures Ġare Ġweighed Ġby Ġquestion Ġand Ġcaption Ġembed ding . ĠThe Ġrest Ġof Ġthe Ġscheme Ġis Ġsame Ġas Ġabove . Ġ( Figure ĠFIG REF 6 ) </s> Mod els Ġ: :: ĠMN - RC NN - W t </s> Same Ġas Ġabove Ġbut Ġwith Ġan Ġadditional Ġlinear Ġlayer Ġapplied Ġto Ġthe Ġdot Ġproduct Ġof Ġcandidate Ġanswer Ġand Ġenc oder Ġoutput , Ġand Ġg ated Ġusing Ġtan h Ġfunction . ĠCompare ĠFigure ĠFIG REF 6 Ġwith ĠFigure ĠFIG REF 6 </s> Mod els Ġ: :: ĠEn semb ling </s> We Ġen semb led Ġfinal Ġlayer 's Ġlog - soft max Ġoutput Ġ- Ġwhich Ġis Ġa Ġdistribution Ġover Ġcandidate Ġanswers Ġfor Ġeach Ġround Ġ( Figure ĠFIG REF 7 ). ĠWe Ġuse Ġthe Ġthree Ġmodels Ġdescribed Ġabove Ġand Ġtake Ġthe Ġmean Ġof Ġthe Ġresults Ġ( we Ġalso Ġtried Ġtaking Ġmaximum Ġof Ġthe Ġresults Ġbut Ġfound Ġmean Ġto Ġperform Ġbetter ). ĠWe Ġalso Ġtried Ġen semb ling Ġa Ġsubset Ġof Ġthe Ġabove Ġthree Ġmodels , Ġbut Ġfound Ġthe Ġcombination Ġof Ġall Ġthree Ġto Ġoutper form Ġthe Ġrest . </s> Exper iments Ġand ĠResults </s> We Ġused ĠPy tor ch ĠB IB REF 5 Ġfor Ġimplementation . ĠIn Ġour Ġexperiments , Ġwe Ġfind Ġthat Ġfine - tun ing Ġinitialized ĠGl ove Ġembed d ings Ġperformed Ġbetter Ġthan Ġfrozen Ġembed d ings . ĠObject Ġlevel Ġrepresentations Ġplay Ġa Ġcritical Ġrole Ġto Ġgenerate Ġa Ġcorrect Ġresponse Ġfrom Ġthe Ġmodel . ĠEventually , Ġwe Ġuse Ġan Ġensemble Ġof Ġall Ġthe Ġmodels Ġdescribed Ġabove Ġfor Ġour Ġfinal Ġsubmission . ĠTable ĠT AB REF 9 Ġsummarizes Ġour Ġresults Ġon Ġvalidation Ġset Ġwhile ĠTable ĠT AB REF 8 Ġon ĠTest - Standard Ġsplit . </s> Conclusion </s> We Ġexperimented Ġwith Ġdiscrim in ative Ġmodels Ġfor Ġour Ġsubmission . ĠObject Ġlevel Ġimage Ġrepresentations Ġgave Ġa Ġhuge Ġupl ift Ġin Ġthe Ġevaluation Ġmetrics . ĠBi - direction al ĠGR Us Ġconstantly Ġperformed Ġbetter Ġthan Ġun i - direction al ĠL ST Ms Ġwith ĠMemory ĠNetworks Ġoutper forming ĠLate Ġfusion Ġenc od ers Ġfor Ġencoding Ġconvers ational Ġhistory . ĠWe Ġeven Ġfound Ġthat Ġfine - tun ing ĠGl ove Ġembed d ings Ġperformed Ġbetter Ġthan Ġtheir Ġcounterparts . ĠOur Ġfinal Ġsubmission Ġis Ġan Ġensemble Ġof Ġthree Ġdiscrim in ative Ġmodels Ġand Ġachieve Ġthe ĠN DC G Ġof Ġ55 . 46 Ġon Ġtest - std .
Introduction Visual dialog BIBREF0 is an interesting new task combining the research efforts from Computer Vision, Natural Language Processing and Information Retrieval. While BIBREF1 presents some tips and tricks for VQA 2.0 Challenge, we follow their guidelines for the Visual Dialog challenge 2018. Our models use attention similar to BIBREF2 to get object level image representations from Faster R-CNN model BIBREF3. We experiment with different encoder mechanisms to get representations of conversational history. Models Common to all the models, we initialize our embedding matrix with pre-trained Glove word vectors of 300 dimensions using 6B tokens . Out of 11319 tokens present in the dataset, we found 188 tokens missing from the pre-trained Glove embeddings, so we manually map these tokens to words conveying semantically similar meaning, e.g. we map over ten variations of the word “yes” - misspelled or not picked up by tokenizer - “*yes", “yesa", “yess", “ytes", “yes-", “yes3", “yyes", “yees", etc. For image features, we extract Faster R-CNN features with ResNet-101 backbone trained on Visual genome BIBREF4 dataset, similar to BIBREF2. We use an adaptive number of object proposals per-image ranging from 10 to 100 generated using a fixed confidence threshold and each object is then associated with 2048-dimensional mean-pooled features using ROI pooling. We use discriminative decoding throughout our models. We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0. Models ::: LF-RCNN Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture. Models ::: MN-RCNN Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6) Models ::: MN-RCNN-Wt Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6 Models ::: Ensembling We ensembled final layer's log-softmax output - which is a distribution over candidate answers for each round (Figure FIGREF7). We use the three models described above and take the mean of the results (we also tried taking maximum of the results but found mean to perform better). We also tried ensembling a subset of the above three models, but found the combination of all three to outperform the rest. Experiments and Results We used Pytorch BIBREF5 for implementation. In our experiments, we find that fine-tuning initialized Glove embeddings performed better than frozen embeddings. Object level representations play a critical role to generate a correct response from the model. Eventually, we use an ensemble of all the models described above for our final submission. Table TABREF9 summarizes our results on validation set while Table TABREF8 on Test-Standard split. Conclusion We experimented with discriminative models for our submission. Object level image representations gave a huge uplift in the evaluation metrics. Bi-directional GRUs constantly performed better than uni-directional LSTMs with Memory Networks outperforming Late fusion encoders for encoding conversational history. We even found that fine-tuning Glove embeddings performed better than their counterparts. Our final submission is an ensemble of three discriminative models and achieve the NDCG of 55.46 on test-std.
Which three discriminative models did they use?
d2a0142150cc3788475572f82458ef88087bd7ac
Which Ġthree Ġdiscrim in ative Ġmodels Ġdid Ġthey Ġuse ?
L F - RC NN , ĠMN - RC NN , ĠMN - RC NN - W t
<s> Which Ġthree Ġdiscrim in ative Ġmodels Ġdid Ġthey Ġuse ? </s> Introduction </s> Visual Ġdialog ĠB IB REF 0 Ġis Ġan Ġinteresting Ġnew Ġtask Ġcombining Ġthe Ġresearch Ġefforts Ġfrom ĠComputer ĠVision , ĠNatural ĠLanguage ĠProcessing Ġand ĠInformation ĠRet ri eval . ĠWhile ĠB IB REF 1 Ġpresents Ġsome Ġtips Ġand Ġtricks Ġfor ĠV Q A Ġ2 . 0 ĠChallenge , Ġwe Ġfollow Ġtheir Ġguidelines Ġfor Ġthe ĠVisual ĠDial og Ġchallenge Ġ2018 . ĠOur Ġmodels Ġuse Ġattention Ġsimilar Ġto ĠB IB REF 2 Ġto Ġget Ġobject Ġlevel Ġimage Ġrepresentations Ġfrom ĠFaster ĠR - CNN Ġmodel ĠB IB REF 3 . ĠWe Ġexperiment Ġwith Ġdifferent Ġenc oder Ġmechanisms Ġto Ġget Ġrepresentations Ġof Ġconvers ational Ġhistory . </s> Mod els </s> Common Ġto Ġall Ġthe Ġmodels , Ġwe Ġinitialize Ġour Ġembed ding Ġmatrix Ġwith Ġpre - trained ĠGl ove Ġword Ġvectors Ġof Ġ300 Ġdimensions Ġusing Ġ6 B Ġtokens Ġ. ĠOut Ġof Ġ113 19 Ġtokens Ġpresent Ġin Ġthe Ġdataset , Ġwe Ġfound Ġ188 Ġtokens Ġmissing Ġfrom Ġthe Ġpre - trained ĠGl ove Ġembed d ings , Ġso Ġwe Ġmanually Ġmap Ġthese Ġtokens Ġto Ġwords Ġconve ying Ġsem antically Ġsimilar Ġmeaning , Ġe . g . Ġwe Ġmap Ġover Ġten Ġvariations Ġof Ġthe Ġword ĠâĢ ľ yes âĢ Ŀ Ġ- Ġmiss pelled Ġor Ġnot Ġpicked Ġup Ġby Ġtoken izer Ġ- ĠâĢ ľ * yes ", ĠâĢ ľ yes a ", ĠâĢ ľ y ess ", ĠâĢ ľ y tes ", ĠâĢ ľ yes - ", ĠâĢ ľ yes 3 ", ĠâĢ ľ y yes ", ĠâĢ ľ y ees ", Ġetc . </s> For Ġimage Ġfeatures , Ġwe Ġextract ĠFaster ĠR - CNN Ġfeatures Ġwith ĠRes Net - 101 Ġbackbone Ġtrained Ġon ĠVisual Ġgenome ĠB IB REF 4 Ġdataset , Ġsimilar Ġto ĠB IB REF 2 . ĠWe Ġuse Ġan Ġadaptive Ġnumber Ġof Ġobject Ġproposals Ġper - image Ġranging Ġfrom Ġ10 Ġto Ġ100 Ġgenerated Ġusing Ġa Ġfixed Ġconfidence Ġthreshold Ġand Ġeach Ġobject Ġis Ġthen Ġassociated Ġwith Ġ2048 - dimensional Ġmean - pool ed Ġfeatures Ġusing ĠRO I Ġpool ing . ĠWe Ġuse Ġdiscrim in ative Ġdecoding Ġthroughout Ġour Ġmodels . </s> We Ġfirst Ġdescribe Ġour Ġmodels Ġindividually Ġand Ġthen Ġthe Ġen semb ling Ġtechnique Ġthat Ġwe Ġemploy . ĠIn Ġthe Ġfollowing , ĠMN Ġdenotes ĠMemory ĠNetworks Ġto Ġencode Ġconvers ational Ġhistory , ĠRC NN Ġsignify ĠR - CNN Ġfor Ġobject Ġlevel Ġrepresentations Ġof Ġan Ġimage , ĠW t Ġrepresents Ġadditional Ġlinear Ġlayer Ġin Ġthe Ġdec oder , Ġand ĠLF Ġa Ġlate Ġfusion Ġmechanism Ġas Ġdefined Ġin ĠB IB REF 0 . </s> Mod els Ġ: :: ĠLF - RC NN </s> Late Ġfusion Ġenc oder ĠB IB REF 0 Ġwith Ġconc aten ated Ġhistory . ĠWe Ġuse Ġtwo - lay ered ĠL ST Ms Ġwith Ġ512 Ġhidden Ġunits Ġfor Ġembed ding Ġquestions Ġand Ġhistory . ĠThe Ġobject - level Ġfeatures Ġare Ġweighed Ġusing Ġonly Ġquestion Ġembed d ings . ĠThe Ġword Ġembed d ings Ġfrom ĠGl ove Ġvectors Ġare Ġfrozen Ġand Ġare Ġnot Ġfine - tun ed . ĠFigure ĠFIG REF 6 Ġgives Ġan Ġoverview Ġof Ġthe Ġarchitecture . </s> Mod els Ġ: :: ĠMN - RC NN </s> Memory Ġnetwork Ġenc oder ĠB IB REF 0 Ġwith Ġbi - direction al ĠGR Us Ġand Ġword Ġembed d ings Ġfine - tun ed . ĠObject - level Ġfeatures Ġare Ġweighed Ġby Ġquestion Ġand Ġcaption Ġembed ding . ĠThe Ġrest Ġof Ġthe Ġscheme Ġis Ġsame Ġas Ġabove . Ġ( Figure ĠFIG REF 6 ) </s> Mod els Ġ: :: ĠMN - RC NN - W t </s> Same Ġas Ġabove Ġbut Ġwith Ġan Ġadditional Ġlinear Ġlayer Ġapplied Ġto Ġthe Ġdot Ġproduct Ġof Ġcandidate Ġanswer Ġand Ġenc oder Ġoutput , Ġand Ġg ated Ġusing Ġtan h Ġfunction . ĠCompare ĠFigure ĠFIG REF 6 Ġwith ĠFigure ĠFIG REF 6 </s> Mod els Ġ: :: ĠEn semb ling </s> We Ġen semb led Ġfinal Ġlayer 's Ġlog - soft max Ġoutput Ġ- Ġwhich Ġis Ġa Ġdistribution Ġover Ġcandidate Ġanswers Ġfor Ġeach Ġround Ġ( Figure ĠFIG REF 7 ). ĠWe Ġuse Ġthe Ġthree Ġmodels Ġdescribed Ġabove Ġand Ġtake Ġthe Ġmean Ġof Ġthe Ġresults Ġ( we Ġalso Ġtried Ġtaking Ġmaximum Ġof Ġthe Ġresults Ġbut Ġfound Ġmean Ġto Ġperform Ġbetter ). ĠWe Ġalso Ġtried Ġen semb ling Ġa Ġsubset Ġof Ġthe Ġabove Ġthree Ġmodels , Ġbut Ġfound Ġthe Ġcombination Ġof Ġall Ġthree Ġto Ġoutper form Ġthe Ġrest . </s> Exper iments Ġand ĠResults </s> We Ġused ĠPy tor ch ĠB IB REF 5 Ġfor Ġimplementation . ĠIn Ġour Ġexperiments , Ġwe Ġfind Ġthat Ġfine - tun ing Ġinitialized ĠGl ove Ġembed d ings Ġperformed Ġbetter Ġthan Ġfrozen Ġembed d ings . ĠObject Ġlevel Ġrepresentations Ġplay Ġa Ġcritical Ġrole Ġto Ġgenerate Ġa Ġcorrect Ġresponse Ġfrom Ġthe Ġmodel . ĠEventually , Ġwe Ġuse Ġan Ġensemble Ġof Ġall Ġthe Ġmodels Ġdescribed Ġabove Ġfor Ġour Ġfinal Ġsubmission . ĠTable ĠT AB REF 9 Ġsummarizes Ġour Ġresults Ġon Ġvalidation Ġset Ġwhile ĠTable ĠT AB REF 8 Ġon ĠTest - Standard Ġsplit . </s> Conclusion </s> We Ġexperimented Ġwith Ġdiscrim in ative Ġmodels Ġfor Ġour Ġsubmission . ĠObject Ġlevel Ġimage Ġrepresentations Ġgave Ġa Ġhuge Ġupl ift Ġin Ġthe Ġevaluation Ġmetrics . ĠBi - direction al ĠGR Us Ġconstantly Ġperformed Ġbetter Ġthan Ġun i - direction al ĠL ST Ms Ġwith ĠMemory ĠNetworks Ġoutper forming ĠLate Ġfusion Ġenc od ers Ġfor Ġencoding Ġconvers ational Ġhistory . ĠWe Ġeven Ġfound Ġthat Ġfine - tun ing ĠGl ove Ġembed d ings Ġperformed Ġbetter Ġthan Ġtheir Ġcounterparts . ĠOur Ġfinal Ġsubmission Ġis Ġan Ġensemble Ġof Ġthree Ġdiscrim in ative Ġmodels Ġand Ġachieve Ġthe ĠN DC G Ġof Ġ55 . 46 Ġon Ġtest - std .