| | --- |
| | license: cc |
| | datasets: |
| | - axgroup/Ranking_TVR |
| | language: |
| | - en |
| | --- |
| | # Video Moment Retrieval in Practical Setting: A Dataset of Ranked Moments for Imprecise Queries |
| |
|
| | The benchmark and dataset for the paper [Video Moment Retrieval in Practical Settings: A Dataset of Ranked Moments for Imprecise Queries](https://arxiv.org/abs/2407.06597). |
| |
|
| | We recommend cloning the code, data, and feature files from the Hugging Face repository at [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking). This repository only includes the code for ReLoCLNet_RVMR. You can download the other baseline models from [XML_RVMR](https://huggingface.co/LiangRenjie/XML_RVMR) and [CONQUER_RVMR](https://huggingface.co/LiangRenjie/CONQUER_RVMR). |
| | |
| |  |
| | |
| | |
| | |
| | |
| | ## Getting started |
| | ### 1. Install the requisites |
| | |
| | The Python packages we used are listed as follows. Commonly, the most recent versions work well. |
| | |
| | |
| | ```shell |
| | conda create --name tvr_ranking python=3.11 |
| | conda activate tvr_ranking |
| | pip install pytorch # 2.2.1+cu121 |
| | pip install tensorboard |
| | pip install h5py pandas tqdm easydict pyyaml |
| | ``` |
| | |
| | ### 2. Download full dataset |
| | For the full dataset, please go down from Hugging Face [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking). \ |
| | The detailed introduction and raw annotations is available at [Dataset Introduction](data/TVR_Ranking/readme.md). |
| | |
| | |
| | ``` |
| | TVR_Ranking/ |
| | -val.json |
| | -test.json |
| | -train_top01.json |
| | -train_top20.json |
| | -train_top40.json |
| | -video_corpus.json |
| | ``` |
| | |
| | ### 3. Download features |
| | |
| | For the query BERT features, you can download them from Hugging Face [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking). \ |
| | For the video and subtitle features, please request them at [TVR](https://tvr.cs.unc.edu/). |
| | |
| | ```shell |
| | tar -xf tvr_feature_release.tar.gz -C data/TVR_Ranking/feature |
| | ``` |
| | |
| | ### 4. Training |
| | ```shell |
| | # modify the data path first |
| | sh run_top20.sh |
| | ``` |
| | ### 5. Inferring |
| | The checkpoint can all be accessed from Hugging Face [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking). |
| | ```shell |
| | sh infer_top20.sh |
| | ``` |
| | |
| | ## Experiment Results |
| | ### Baseline |
| | The baseline performance of $NDGC@40$ was shown as follows. |
| | Top $N$ moments were comprised of a pseudo training set by the query-caption similarity. |
| | |
| | | **Model** | **Train Set Top N** | **IoU=0.3** | |**IoU=0.5** | |**IoU=0.7** | | |
| | |----------------|---------------------|--------------|--------------|--------------|--------------|--------------|--------------| |
| | | | | **Val** | **Test** | **Val** | **Test** | **Val** | **Test** | |
| | | **XML** | 1 | 0.1077 | 0.1016 | 0.0775 | 0.0727 | 0.0273 | 0.0294 | |
| | | | 20 | 0.2580 | 0.2512 | 0.1874 | 0.1853 | 0.0705 | 0.0753 | |
| | | | 40 | 0.2408 | 0.2432 | 0.1740 | 0.1791 | 0.0666 | 0.0720 | |
| | | **CONQUER** | 1 | 0.0952 | 0.0835 | 0.0808 | 0.0687 | 0.0526 | 0.0484 | |
| | | | 20 | 0.2130 | 0.1995 | 0.1976 | 0.1867 | 0.1527 | 0.1368 | |
| | | | 40 | 0.2183 | 0.1968 | 0.2022 | 0.1851 | 0.1524 | 0.1365 | |
| | | **ReLoCLNet** | 1 | 0.1533 | 0.1489 | 0.1321 | 0.1304 | 0.0878 | 0.0869 | |
| | | | 20 | 0.4039 | 0.4031 | 0.3656 | 0.3648 | 0.2542 | 0.2567 | |
| | | | 40 | 0.4725 | 0.4735 | 0.4337 | 0.4337 | 0.3015 | 0.3079 | |
| | |
| | |
| | ### ReLoCLNet Performance |
| | |
| | | **Model** | **Train Set Top N** | **IoU=0.3** | |**IoU=0.5** | |**IoU=0.7** | | |
| | |------------|---------------------|--------------|--------------|--------------|--------------|--------------|--------------| |
| | | | | **Val** | **Test** | **Val** | **Test** | **Val** | **Test** | |
| | | **NDCG@10** | | | | | | | | |
| | | ReLoCLNet | 1 | 0.1575 | 0.1525 | 0.1358 | 0.1349 | 0.0908 | 0.0916 | |
| | | ReLoCLNet | 20 | 0.3751 | 0.3751 | 0.3407 | 0.3397 | 0.2316 | 0.2338 | |
| | | ReLoCLNet | 40 | 0.4339 | 0.4353 | 0.3984 | 0.3986 | 0.2693 | 0.2807 | |
| | | **NDCG@20** | | | | | | | | |
| | | ReLoCLNet | 1 | 0.1504 | 0.1439 | 0.1303 | 0.1269 | 0.0866 | 0.0849 | |
| | | ReLoCLNet | 20 | 0.3815 | 0.3792 | 0.3462 | 0.3427 | 0.2381 | 0.2386 | |
| | | ReLoCLNet | 40 | 0.4418 | 0.4439 | 0.4060 | 0.4059 | 0.2787 | 0.2877 | |
| | | **NDCG@40** | | | | | | | | |
| | | ReLoCLNet | 1 | 0.1533 | 0.1489 | 0.1321 | 0.1304 | 0.0878 | 0.0869 | |
| | | ReLoCLNet | 20 | 0.4039 | 0.4031 | 0.3656 | 0.3648 | 0.2542 | 0.2567 | |
| | | ReLoCLNet | 40 | 0.4725 | 0.4735 | 0.4337 | 0.4337 | 0.3015 | 0.3079 | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | ## Citation |
| | If you feel this project helpful to your research, please cite our work. |
| | ``` |
| | |
| | ``` |
