Controlling concurrent package distribution in SCCM

I normally only write blog posts about subjects that aren’t covered particuarly well on the web or that seem, on the whole, to be misunderstood and I think this area is certainly one of them.

Recently we had some issues on one of our primary SCCM site servers, that has around 70 secondary sites, which caused a huge backlog of packages, over 3000, waiting to install. Of the many different packages that were waiting to be sent, one package was over 10GB in size and two were nearly 4GB. As you can imagine sending this much data, to 70 different sites, was taking a long time and as new packages were being distributed all the time the queue was only getting longer. Now obviously if we had left the sending capacity as default it would have eventually cleared the backlog, once the large packages had finally sent, but it was starting to affect operations as software waiting to go live couldnt, due to waiting for distribution. So we decided to crank up sending capacity.

Now I’ve always assumed the way to do this was from Site Settings > Component Configuration > Software Distribution > Distribution Point tab and there is the “Concurrent distribution settings” options, which allow you to change “Maximum number of packages” and “Maximum threads per package”.

What I have done in the past is put the “Maximum number of packages” to 7 (which is the maximum) and set “Maximum threads per package” to 10, as there is no real guidance as what this should be set at (the max is 50). Now at this point I just assumed the amount it sent would increase and think nothing else of it but this time I wanted to monitor what was being sent and to what secondary sites.To do this I used a small application that a friend of mine developed called Sender Analyser (http://www.danrichings.com/?p=90) that reads the sender.log and inteprets it to give you a list of active sending jobs. Its much easier than trying to read the sender.log, I promise you!

The results of the Sender Analyser showed the following:

and the sender.log showed this:

Both show that the Primary server was only sending 5 (out of a maximum of 5) packages to the secondary sites at any one time, even though I set both “Concurrent distribution settings” values to 7 and 10. This started me thinking whether these settings I’d changed had anything to do with site to site distribution of packages at all but were actually settings designed to control the distribution of packages to local or remote DP’s of its own site only.

To check I decided to have a look at the distmgr.log of my primary site and the first thing I noticed was this:

So the distmgr.log was allowed to process up to 7 packages at a time, exactly what I had set the”Maximum number of packages” value to. To test my theory I changed the same setting to 6 and the distmgr.log responded by showing “Used 0 out of 6 allowed…” exactly like I thought it would.

So what does this mean? Well the SMS_DISTRIBUTION_MANAGER component is mainly used to uncompress the compressed package files that are sent to it from its parent site into proper package format that can be downloaded by the clients. If the primary or secondary site has multiple servers setup as DPs in its own site, either locally or remote, this is the component in charge of copying these package files from the site server to the DP. The “Concurrent distribution settings” are the way you control how many packages and how many threads (I’m not sure if threads apply to concurrent number of servers or processor threads) it can handle at one time. What seems clear to me is that it doesn’t affect the way that packages are sent from a parent site to a child site.

So, you may be asking, how do you increase the amount of concurrent packages a parent site can send to its child sites from the, apparent, default of 5? Simple. All you need to do is change the properties for your Sender.

By going to Site Settings > Senders > Standard Sender > Advanced tab. Here is the “Maximum concurrent sendings” options, which allow you to change the maximum number of objects that can be sent to “All sites” or “Per site”:

As you will see the default for “All sites” is 5 and in our previous sender.log we saw the max sending capacity was also 5. Coincidence? No I didnt think so either, so I started increasing the “All sites” value slowly and seeing if it matched what the sender.log told me, eventually increasing it to 220. The sender.log showed this:

This quite clearly showed that the values in the “Maximum concurrent sendings” setting and the sender.log matched and this was backed up by the Sender Analyser.

The sender was now able to send nearly 50 times as many packages as the default amount and its fair to say that my package backlog didnt take long to clear! I eventually settled on “All sites” to 200 and “Per site” to 20 but the all sites capacity rarely got above 100, I guess there is something else causing the bottleneck, still that was plenty enough. Please note that I was able to use these high values due to the spec of my server, my LAN and WAN capacities, DON’T ASSUME YOU CAN USE THE SAME. I strongly recommend speaking to your network team first and slowly raising the values until you are happy.

So there we have it, if you are looking to increase the capacity to send more packages to your child sites, make sure you change the right setting 🙂

 

Advertisements

5 Responses to Controlling concurrent package distribution in SCCM

  1. EdW says:

    Thanks. That’s a great tip. I made the suggested changes and noticed in the Sender Analyser (great tool too!) after a short while that the number of packages increased to the total number of Secondary sites we have (9) from the default 3. We’re using bandwidth limiting so the data transfer will remain throttled, but at least we can send simultaneously to all the sites from the Primary site. We’ll be growing to over 50 sites, so we definitely needed this setting change.

    Your article helps decipher what’s going on in the logs as well.

    For some reason, on a few packages, the total size and amount remaining report as unavailable, and on those same packages the Package ID column displays the ID, but for the others, only the Request ID displays that ID but does report the total size and amount remaining. Not a big deal.

  2. Joshua Holden says:

    This is great to know. Thanks so much for writing this up. I do have a question though.

    I have one site server and about 150+ DPs(I did NOT design this). The “concurrent distribution settings” is in charge of how many packages the site will copy out to the DPs. Right?
    Then does the “maximum concurrent sendings” set how many DPs the site will copy to at once or is only for sites, as in site servers? Which setting will help me increase the number of DPs the site will push to at once?

    Thanks for any help. And great article!

    • nivor says:

      Hi Joshua,

      The “Maximum number of packages” setting under “concurrent distribution settings” sets the maximum number of packages that can be decompressed to DPs at one time but that could be to 1 DP or multiple DPs. I don’t think it matters as long as the number of concurrent decompressions doesn’t go above the value.

      I believe the “maximum concurrent sendings” setting relates to site to site comms (package distribution to lower secondaries or normal traffic) and package distribution to clients, if the site or DP you set this on has any machines in its boundaries. I don’t think (but I’m not 100% sure) that it relates to package decompression to DPs (which is your scenario), as I think this is handled with the distmgr component (managed by the “concurrent distribution settings”) rather than the sender component.

      If I were you I would test it by changing the “maximum number of packages” setting to say 10 and then deploy a large package to more than 10 DPs and look at the distmgr.log file on the site you changed the setting. The log should state “Used 10 out of 10 allowed processing threads” and you should be able to see the servers the files are being sent to you. Once you are sure it is decompressing 10 packages to 10 DPs at once, change the setting to what you think is a sensible number.

      I’d be interested to hear the results, so please post how it goes! 🙂

  3. Nagender says:

    Hi, When I am using Sender Analyser, i am getting ” cannot locate sender.log. Please check the server and site code” error. Any idea?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: