experiment.md

在Rotten_Tomatoes上面做了实验，将small, base, large三个配置在卷积方法和原始transformer进行了比较

文本分类数据集（测试集准确率，5个种子取平均）

Rotten_Tomatoes

• base_transformer: 75.30 ± 0.66%
• conv_base_transformer: 75.51 ± 1.36%

imdb

• base_transformer: 84.51 ± 0.40%
• conv_base_transformer: 83.82 ± 0.51%

ag_news

• base_transformer: 91.99 ± 0.11%
• conv_base_transformer: 91.62 ± 0.20%

20_newsgroups

• base_transformer: 60.63 ± 0.53%
• conv_base_transformer: 64.67 ± 0.36%

GLUE 数据集（验证集准确率）

glue_cola

• transformer: 69.23 ± 0.14%
• conv_transformer: 69.17 ± 0.05%

glue_wnli

• transformer: 56.34 ± 0.00%
• conv_transformer: 56.34 ± 3.45%

glue_mnli

• transformer: 55.80 ± 0.21%
• conv_transformer: 59.72 ± 0.29%

glue_mrpc

• transformer: 70.58 ± 0.14%
• conv_transformer: 70.28 ± 0.38%

glue_qnli

• transformer: 61.01 ± 0.48%
• conv_transformer: 61.34 ± 0.35%

glue_qqp

• transformer: 77.65 ± 0.12%
• conv_transformer: 79.27 ± 0.09%

glue_rte

• transformer: 53.58 ± 1.37%
• conv_transformer: 53.21 ± 1.04%

glue_sst2

• transformer: 81.49 ± 0.63%
• conv_transformer: 82.20 ± 0.41%

kernel	original	5	7	9	11
Rotten_Tomatoes	75.18%	75.29%	77.06%	73.29%	77.06%
imdb	84.50%	83.65%	85.47%	85.24%	85.70%
ag_news	91.61%	91.95%	92.28%	92.38%	92.13%
20_newsgroups	60.36%	61.14%	62.94%	60.91%	62.29%

Rotten_Tomatoes

• 原始准确率: transformer 75.18% -> test_transformer 77.06%
• Mask(0.05): 74.00% -> 76.00%
• Mask(0.1): 74.00% -> 75.06%

imdb

• 原始准确率: transformer 84.50% -> test_transformer 85.47%
• Mask(0.05): 84.63% -> 85.26%
• Mask(0.1): 84.05% -> 84.75%

ag_news

• 原始准确率: transformer 91.61% -> test_transformer 92.28%
• Mask(0.05): 91.11% -> 91.72%
• Mask(0.1): 91.09% -> 91.13%

20_newsgroups

• 原始准确率: transformer 60.36% -> test_transformer 62.94%
• Mask(0.05): 59.32% -> 60.83%
• Mask(0.1): 58.74% -> 59.84%

CIFAR-10

• 原始准确率: vit 91.48% -> test_vit 93.28%
• Gaussian(0.05): 40.10% -> 43.82%
• Gaussian(0.1): 38.98% -> 42.50%
• Gaussian(0.2): 33.84% -> 37.72%
• Salt-Pepper(0.1): 48.41% -> 55.40%
• Uniform(0.1): 39.93% -> 43.53%
• CvT(kernel_size=3): 89.00%

CIFAR-100

• 原始准确率: vit 67.30% -> test_vit 70.08%
• Gaussian(0.05): 17.46% -> 22.07%
• Gaussian(0.1): 14.25% -> 19.40%
• Gaussian(0.2): 8.59% -> 13.02%
• Salt-Pepper(0.1): 13.39% -> 18.27%
• Uniform(0.1): 17.33% -> 21.43%
• CvT(kernel_size=3): unknown

FashionMNIST

• 原始准确率: vit 94.25% -> test_vit 94.56%
• Gaussian(0.05): 66.77% -> 74.55%
• Gaussian(0.1): 67.28% -> 74.48%
• Gaussian(0.2): 67.14% -> 75.16%
• Salt-Pepper(0.1): 84.61% -> 85.54%
• Uniform(0.1): 67.03% -> 74.50%
• CvT(kernel_size=3): 94.12%

SVHN

• 原始准确率: vit 97.04% -> test_vit 97.03%
• Gaussian(0.05): 72.91% -> 73.54%
• Gaussian(0.1): 72.01% -> 73.19%
• Gaussian(0.2): 68.15% -> 70.57%
• Salt-Pepper(0.1): 78.80% -> 77.47%
• Uniform(0.1): 72.86% -> 73.34%
• CvT(kernel_size=3): 96.87%