I did the following.
1) Changed the IP of my xbox. There are different methods of doing this. I achieved it by unplugging my modem for 10 minutes. Setting up a static ip may be preferrable but will require additional tech knowledge. I have set up a static ip before but chose not to at this point in time.
2) I placed my xbox in a DMZ. This can typically be done by accessing your modem gateway page. It is relatively easy and only requires you to enter your ipv4 and where applicable ipv6 ip address for your xbox. Both of these can be found in the network settings page of your xbox.
3) I turned off upnp. There should be an on / off button to this in your modem gateway page.
I was disconnecting upwards of 10-15 times a week for 2 weeks. Including 4 times in a day. All were server disconnects. I have not had a server dc in upwards of 100 games since making these changes.
A dmz basically opens all traffic to your xbox. Upnp opens up ports upon receiving the request. Port forwarding allows you to select which ports to open. Ill post a thread on this in the GD as well.
Note: These changes have significant security risks / flaws. You need to do your own research to determine if the risks are acceptable. The same things can be done on a ps4 just you will obtain your ip in a seperate settings page. Additionally with time depending on your modem / provider your ip may change. This may require you to update your xbox ip within the dmz.
Typically the method I used is for obtaining an open nat but it seems to have resolved my server disconnect issues.@Joestoe