Two Successive Linear Layer is same with one linear layer.

[developer0hye]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi!, @glenn-jocher and @dingyiwei !

I am trying to improve the performance of object detection model with transformer layer. And I found that @dingyiwei already experimented it on this project.

I have one question about your method.

yolov5/models/common.py

Line 66 in 79bca2b

x = self.fc2(self.fc1(x)) + x

Why do you design the transformer layer with two successive linear layer?

Two Successive Linear Layer is same with one linear layer theorectically.

Additional

No response

[glenn-jocher]

@developer0hye that's a good question, but I can't answer that myself since @dingyiwei is the C3TR author in PR# #2333.

I know 2 convolution2d layers in series are reducible to 1 if it were not for the activation layer in between, does the same hold true for fully connected layers with no activation in between?

[developer0hye]

@glenn-jocher

If the kernel size of conv2d layer is one, your statement is true!

[dingyiwei]

Hi @developer0hye , thanks to point out this issue! It should be an activation between the two successive linear layer. I'll run experiments on that and post results here.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[mx2013713828]

Hi @developer0hye , thanks to point out this issue! It should be an activation between the two successive linear layer. I'll run experiments on that and post results here.

I'm waiting your experiments.

[dingyiwei]

In this comment I mentioned why I removed the activation function.
I have to find a new computing platform and rerun my experiments because I left the lab recently before.