Windows Pagefile Done Right

Over the years there has been a lot of information on configuring the pagefile, paging has gone through the evolution on what size and configuration it should be. With virtual machine RAM allocation going higher and higher and a lot of guidance from the likes of Citrix and VMware to potentially build VMs with 64GB+ RAM. Paging is a very interesting topic as setting the pagefile size too high, wastes disk space; removing the page file entirely is bad as well as Windows needs to have a pagefile even though we do not want the system to page. Not having a pagefile equates Windows complaining about not having enough virtual memory if the actual memory is fully allocated. While no one wants to page and systems should be built to minimize paging aka allocating enough memory to let folks operate their applications without paging but systems should be configured to allow some sort of paging as a "use in case" measure. Lastly the out of box configuration of a system managed pagefile is also a bad idea.

Setting the pagefile to system managed in an enterprise type of environment is a bad practice. The reason why? If there are any monitoring tools a lot of monitoring tools track pagefile utilization and it is typically tracked as a percentage. If the pagefile is set to system managed and the minimum/maximum are not set, it can result in monitoring tools reporting excessive pagefile utilization.

Setting the pagefile with the old school approach of setting the pagefile 1.5x the size of the memory results in this configuration. We will hear a lot of complaints about wanting to capture some sort of dump for Microsoft to "analyze". When was the last time Microsoft successfully analyzed a dump with meaningful results? For me it has never happened.

Here is what the drives would look like if we for instance allocated 64GB of RAM to a VM and are using the old school approach of 1.5x RAM for the pagefile:

Here is the error in the eventlog when the pagefile is completely eliminated as virtual memory is low:

What is the happy medium? Setting the pagefile to either 4096MB/4GB for both the minimum and maximum on single user operating systems and 8192MB/8GB for both the minimum maximum on multi-user operating systems.

Why is this the happy medium? I checks all of the boxes as it allows for Windows to have a pagefile if needed, it allows for a minidump to be configured/captured/analyzed in Windows if needed and it is not too large that we feel like we are wasting space.

What about memory dumps? Well if there is a need for some sort of memory dump to be captured and analyzed, Windows can be configured to generate a minidump. There are plenty or articles/blog posts out on the internet on how to configure Windows to produce a minidump.

If you have any thoughts, we would like to hear from you in the comments.

Johnny @mrjohnnyma

Don't Treat Your Virtual Desktop Security Like Your Physical Desktop Security

 

This blog post complements a previous blog post I wrote a bit ago talking about not using your physical image in your virtual environment. To check out that one please refer here. 

Are you using anti-virus, anti-malware, data loss prevention (DLP) software or the like on your virtual desktops? Are you treating them the same as you would on a physical desktop? If the answer was yes to both this is the blog for you. If you are not using anti-virus on your virtual desktop that is a whole other conversation and potential can of worms that needs to be addressed. When running any of the various security tools out there we need to consider the need to configure the proper exclusions to ensure everything runs properly and the users are not getting performance degradation because these exclusions are missing. I see this all of the time that folks are not properly implementing the proper security tool exclusions into their virtual desktop images or they configure the exclusions in the various consoles and they can be shown when asked but machines are not landing in the proper container to actually get the exclusions. I recently was working with a customer that was suffering severely slow/long application launch times in applications such as Outlook, Teams, OneDrive, etc... Upon examining they were capturing things like the Outlook OST, Teams Cache and OneDrive cache into virtual disks stored on a network share as VHD/VHDX files. When users would log onto a virtual desktop and these virtual disks mounted they were being actively scanned by anti-virus and when the Outlook, Teams and OneDrive clients were trying to read the data on the virtual disks performance was hampered because of the scan.

The above not only just applies to non-persistent desktops but to fully persistent desktops as well. I know I will get the response of "aren't persistent desktops the same as physical desktops?" The answer is yes and no. While anything that gets written to the disk is fully stateful and there may or may not be any profile management happening on these desktops. There are still the core virtual desktop components installed to deliver folks the remote display capability with the requisite virtual channels to allow for things like audio/video redirection and offloading. Therefore we still need the proper security tool exclusions to ensure everything is as optimized from the security perspective as possible.  In addition to this with modern-day laptops/desktops, there are potentially a lot more resources in terms of CPU and RAM compared to what is allocated on the virtual desktop side. So, an un-optimized anti-virus/anti-malware utility's impact on the physical side may not be as noticeable.

Long story short, spending a little bit of extra effort to make sure security tools are configured properly will save the headaches of dealing with complaints about bad experience. Just as I said in the previous blog of the common adage "you can't build a house on a bad foundation." This holds very true on this conversation as well.

If you have any thoughts, we would like to hear from you below in the comments.

Johnny @mrjohnnyma

Replacing a Self-Signed Certificate on vCenter 7.x +

Purpose:

Demonstration on how to replace the self-signed certificate on VMware vCenter.

Introduction:

Having valid certificates is not only crucial today and going forward, it has been crucial for the last few years as well. Having valid certificates not only ensures that a certain security posture being maintained, it removes any unsightly certificate warnings that make various products unfriendly to use for the administrators/engineers/architects.

I recently made a transition from Nutanix Community Edition (CE) to VMware vSphere in my home lab due to upgrade issues with the most recent release of CE. VMware vSphere 7.x and above resolved an issue where the NIC in an Intel NUC 10 was not detecting during installation and the driver needed to be sideloaded before CE could be installed. This is a continuation of my blog series where I take a focus in on security from a virtualization standpoint. Here is a similar themed blog about how to replace the self-signed certificate in Nutanix Prism Element and Prosim Central.

Today we will talk about how to replace the certificate on vCenter and how significantly easier it has become to do so. Before I start, I am going to preface this that process only applies to VMware vCenter 7.0 and above at the time of this writing. If folks are still running a vCenter 6.5 or 6.7 this will not work there as the process is completely different. Also this not only affects Citrix, it affects VMware Horizon and any other solutions that integrate into vCenter.

How many of us have in the past or even today check the box on this message to acknowledge and trust the self-signed certificate in an on-prem or cloud based full Citrix Studio?

Most of us probably click through it without second thinking why  the warning applies or also just wave it off as “that is not my problem and it is the vSphere team’s problem”. While it may be the vSphere teams problem, security should be a concern from all IT folks as there are always ways that system compromises can easily be fixed if there was a security first mentality. In addition to this, replacing the certificate will remove the warning from vCenter when folks use the vCenter web console. 

In vCenter 7.0 and above it is very easy to replace the certificate so that the warning never even pops up when establishing the Hosting connection string from Studio. 

Configuration Steps:

First we will need to create a certificate, in my case I will be using a domain certificate authority (CA). A certificate from a 3rd party well trusted CA can also be configured in this manner as well. 

I find it easier to generate the CSR on the vCenter and later will have some interesting issues from generating the CSR elsewhere.

Go to vCenter and login as administrator@vsphere.local (this is the only account that has permissions to change the certificate management) On the Top, go to Menu -> Administration

On the left pane -> Click Certificate Management

Under Actions -> Click Generate Certificate Signing Request (CSR)

Fill out the information appropriately -> Click Next

Copy or Download the CSR -> Click Finish

Open a browser and go to https://domainca.fqdn.com/certsrv replacing with your domainca FQDN. In my case it is domain1.domain.lab. -> Click Request a Certificate

Click Advanced Certificate Request

Click Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file

Copy and paste the contents of the CSR file generated earlier into the large field -> Select the appropriate certificate template -> Click Submit

After submitting the certificate may be pending if the CA is configured for approval (as such in my lab). Get the proper approval to issue the certificate

After approval go back to https://domainca.fqdn.com/certsrv -> Click View the Status of a Pending Certificate Request

Click on the Request from earlier -> Click on the Request

Select Base64 encoded -> Download the Certificate

Save with to a location where it can be accessed with an appropriate name –> Click Save

The domain CA’s root and intermediate certificates are required to be exported as .cer as well. In my case, these can be found on the domain controller under Certificate Manager for the Local Machine -> Trusted Root Certificate Authorities Certificates.

Back on vCenter -> Administration -> Certificate Management we need to import the Root and intermediate certificates so that the cert is trusted. -> Click Add

Browse to the root cert -> Click Add

After adding, there are now multiple Trusted Root Certificates

For the Machine Cert section Click Action -> Import and Replace Certificate

Select Replace with external CA certificate where CSR is generated from vCenter Server (private key embedded) as the CSR was generated on the vCenter -> Click Next

On the first field -> Click Browse File and select the certificate that the Domain CA issued. On the second field -> Click Browse File and select the domain CA root certificate that was exported. If there are both root and intermediate certificates they may need to be combined in notepad –> Click Next

vCenter Services will automatically restart which will take a few minutes. It is common to get this message as services are restarted.

When vCenter is back and ready log back in and go to the Certificate Management section. The Machine cert should have an updated expiration date. Track that date and make sure to repeat the process again before the certificate expires to ensure everything continues to run smoothly for any services that integrate with vCenter.

There also are no longer certificate warnings when going to the vSphere web client and when the certificate is viewed, it is the appropriate certificate

The Hosting section in Studio connects to vCenter without a warning now as well.

If you tried to generate the CSR outside of vCenter and went through the process of generating the certificate. You could get this error like I did. There really isn’t a reason why the character was invalid but this is why I recommend generating the CSR on vCenter.

Conclusion:

VMware has made it significantly easier to replace the certificate in vSphere 7.x then it was in 6.x. It makes it almost a no-brainer to do this in my opinion. We didn't need to incure any additional costs as the certificate was generated from a domain CA, but this process would work if you need to get a signed certificate from a third party CA. If we take an overall approoach of focusing in on security in each layer of the infrastructure, we significantly improve the security posture of the entire environment and eliminate as many security flaws in the environment as possible.

We would like to hear from you so feel free to drop us a note if you have any questions.

Johnny @mrjohnnyma

8.7.3

Don't Use Your Physical Image in Your Virtual Environment

Are you using SCCM, WDS or other deployment tools or have been asked to when deploying your virtual desktops or virtual application servers? If so, there can be some serious issues with this. I am often asked about by folks wanting to deploy Citrix or VMware Horizon images using the same image that is used for physical endpoints. Not only is this a bad idea, it can present performance ramifications and also make it so that best practices are not followed.

I always have been a believer that hand building the operating systems for virtual desktops and application delivery servers is the best approach because it ensures we know what went into the image. I understand the grips of manually installing the applications and the extra work but the extra work now can save a lot of headaches later and the reason of "this is how we build out images" is not a good enough reason to justify using the same image in the virtual environment.  Often and in most cases the deployment person and the virtual desktop environment are not the same person. They build images on physical endpoints or on a completely different hypervisor, they never optimize the image and just let things fly. Since these are physical endpoints they have dedicated hardware and rarely if ever do they experience any issues from being unoptimized. In the datacenter, on a virtual desktop or an application delivery server which share host resources with other virtual machines we need to optimize things as much as possible.

Here are two examples of recent environments where there were issues with using SCCM to deploy the same image as physical endpoints:

1. First was in the medical field and the customer wanted to move from persistent Windows 10 desktops to pooled non-persistent virtual desktops as the administrative overhead of having a persistent desktop and having to administer the desktops with deployment tools was not feasible. Also, when presented with justifying the need of having a persistent desktop pool and having the response be “that is how we have deployed it before” there really was no reason to have it. When it came time to build the Windows 10 non-persistent image, the customer completely disregarded my suggestion on building the Windows 10 base image by hand and used WDS to deploy the “standard” image that is deployed on physical endpoints. The end result was that a known bug in the image in which the start menu stopped responding to left clicks. This bug also existed on physical endpoints but was hacked around by copying profiles over the default profile but when this was done on the non-persistent desktop image, it caused Citrix Profile Management to create temp profiles on each login. After countless days of the customer trying to remediate this, the only successful way to do so was to break out the iso and install the operating system by hand and manually installing the applications and everything is functioning correctly. 
2. A second example of this was a large law firm migrating from an on-prem Citrix environment to VMware Workspace ONE. When it came time to build their images for the RDS Linked Clone pool they stressed a need to use an existing task sequence that was built for Windows 10 and force it to target a Window Server 2016 operating system. The issue here is that applications were installed before the RDS Session Host role was installed afterwards. It has commonly been a known and best practice for RDS Session Hosts servers that the RDS Session Host role to be installed prior to installing applications due to the need to potentially capture applications settings into the RDS shadow key. In this environment, there are small abnormalities in application behavior even today due to the incorrect installation sequence.

Long story short, when building the images for your virtual desktops and application delivery servers be careful how you approach this. As the common adage is "you can’t build a house on a bad foundation" and doing things incorrectly could lead to a bad user experience.

Johnny @mrjohnnyma

Citrix MCS - Automate Master Image vCenter Update

Purpose:
This post is to share a Citrix MCS / VMware PowerShell script I created over the last year or so to automate the process of renaming a master image (Win2K16 or W10), converting it to a template, cloning the template and creating a snapshot of the clone in preparation for MCS updates.

You can use this script if you have a single datacenter or if you have multiple. It's relatively easy to modify it so the master image is copied to multiple vCenter datacenters/clusters and follows the same process.

I currently have 9 different datacenters/clusters running the same master image. I just maintain the core master image in the primary datacenter, perform updates, shutdown, run this script and once the process is done I can update each machine catalog one by one from the same master image.

Download the script from Citrix ShareFile

SageLike Post ID: SL0023

Citrix PVS - Automate PVS Target Device and VM Creation

Purpose:
This post is to share a Citrix PVS / VMware PowerShell script I created over the last year or so to automate the creation of additional VM's in VMware and create the PVS devices, assign them a PVS image, a PVS site, add them to the domain and then add the vDisk to the new devices. I have transitioned jobs since I first wrote this script so I haven't updated it in a few months but figured it would be helpful to some out there

SageLike-PVS-VMware-Script


SageLike Post ID: SL0020

Applies to:
Citrix Provisioning Server
Citrix XenApp
Citrix XenDesktop
VMware ESXi

VMware ESXi 5.5 - Attach 4K RDM Error. Failed to Start Virtual Machine

Purpose:
This post explains why when adding a 4K LUN as an RDM to a VM doesn't work

Symptom:
When a 4K block sized LUN is added to a virtual machine on an ESXi 5.5 environment you get the below error when trying to boot the VM.  



Resolution:
Either recreate the Volume/LUN to one that is 512, upgrade vCenter and your ESXi hosts to version 6 or higher. In our case we were trying to P2V the OS of a SQL server and then move over the direct attached LUN to the VM. We had to back out our change since we were still on ESXi 5.5.


Cause:
As designed. VMWare doesn't support 4k LUNs until after ESXi 6. 

SageLike Post ID: SL0017

Applies to:
ESXi 5.5 and earlier

References:
https://gruffdba.wordpress.com/2015/08/02/4k-logical-block-size-size-fails-on-vmware/  
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2091600

Kenneth is Speaking at BriForum Denver

I sat down with one of my coworkers, Kenneth Fingerlos, to discuss his upcoming speaking engagement at the BriForum conference in Denver, Colorado on July 20th.  Our brief conversation covered the details of his session, "vSGA, vDGA, vGPU, and Software - When and Why", his background in the industry, and what gets him excited in the technology space right now.

"BriForum excites me because it is everybody" - Kenneth

Me: Kenneth, can you tell me a little bit about your industry experience?

Kenneth: So, after college I took a left turn in my career path and went into corporate IT for ten years.  Various positions: desktop management, server management, data center.  Various kinds of things.  After ten years of that I decided I didn't care for IT management and tried to correct the course change and landed in consulting.  I've been doing IT consulting for about the last ten years around storage, data management, virtualization of various types, and building up my skill sets trying to help customers solve problems.

Me: Great, great.  So have you been to BriForum before? 

Kenneth: I have not been to BriForum.  This will be my first year.

Me: What attracted you to BriForum? 

Kenneth: I'm excited.  The whole idea of a conference that has some size to it and is established that is not tied to a specific vendor is just exciting, right?  You go to a Cisco conference and it is all about what is the latest widget from Cisco.  Cisco can do no wrong.  You find the same thing if you go to, you know, Dell World.  Dell is perfect.  Whatever Dell has got going is awesome and whatever everyone else has is garbage.  BriForum excites me because it is everybody. It is a marketing company--a media company that puts on the conference as opposed to a product manufacturer. insert kennth photo here

Me: So what will you be discussing at BriForum? 

Kenneth: I'm discussing a topic that is near and dear to my heart which is the idea of virtualized graphics.  Taking things we do everyday in the physical world with physical PCs and trying to bring this into this virtualized environment.  Things like disaster recovery, security, flexibility.  You know, the physical world is pretty restricted.  Graphics have always been one of these things that is hard and is difficult.  Technology is evolving and has advanced dramatically over the last couple of years in terms of what we can do.  But there is also a lot of complexity and a lot of information and I find my customers have a lot of confusion about what they can and can't do.  What works, what doesn't work.  My session is all about trying to bring some clarity to that area.

Me: Ok, so I am going to open this up a little bit and say maybe don't limit this to just the enterprise world but what is the technology you are most excited about right now? 

Kenneth: The technology I am most excited about right now....I think the stuff that is most exciting is really this idea of graphics virtualization.  I mean, so many things go into a user experience, right?  And all of the traditional things that you think about: servers, storage, memory, CPUs--graphics is part of that.  Remoting protocols, right?  What's going on with actually getting that content delivered to a user.  Networking, right? 3G, 4G networks and starting to think about what's next, what's beyond 4G.  These are huge enablers to let people consume and develop content in ways that have never been envisioned before.  Letting you take that stuff to the cloud, to the remote data center, and access it from anywhere.  I've been sitting on top of a mountain in my 4x4 holding a virtual desktop, just because I'm a geek and into this stuff, but yes--I can access that app, whatever it is, from a mountain top in the middle of nowhere.  That's cool stuff. And it's all about enabling people to work and function in ways they've never been able to before.  That excites me.

Me: Very cool.  Well, looking forward to seeing your session at BriForum!  Until next time.

As I wrote about earlier, BriForum Comes to Denver, and I am excited to have such a great event in my backyard.  If you are going to be at BriForum or just have general questions about Denver, reach out to either @kfingerlos or myself (@sagelikebrian) and let's catch up.


Brian @sagelikebrian

BriForum Comes to Denver

IT conferences are a great way to catch up on what is new, take classes, and network with peers in the industry. I have been lucky enough to attend great shows like Citrix Summit and Synergy as well as VMware VMworld over the years. The conference for me that always fell just out of reach was BriForum. This year it is all going to change. I am more than a little excited that one of the world's premier IT conferences has chosen Denver, Colorado for this year's US location. BriForum is an independent conference that provides vendor-neutral perspective on current and emerging technologies and services.

 
Check out this year's list of sessions: http://www.brianmadden.com/blogs/gabeknuth/archive/2015/03/09/check-out-the-list-of-sessions-for-briforum-denver-2015-july-20-22.aspx

If you have a keen eye, you may have noticed a third of the way down the list a special session, "vSGA, vDGA, vGPU, and Software - When and Why", being presented by our very own expert speaker Kenneth Fingerlos (@kfingerlos).

Kenneth will be talking about the new graphics intensive workloads that are possible in VDI thanks to highend GPUs from NVIDIA. He will specifically be digging into the different methods you can use to virtualize the GPU and when and why you would want to choose each method. I promise you this will be a deep technical dive preparing you for your next graphics intensive virtual desktop project.  Come join us at BriForum 2015 if you would like to learn more about solutions from Citrix, VMware, Microsoft and much more.

Brian Olsen (@sagelikebrian)

Dude, where's my admin?

My standard joke for a long time has been that I get paid to click “Next” for a living.  2011 will go down as the year that I stopped getting paid to click “Next” and started getting paid to read admin guides.  I am certainly not alone.  Guys and girls around the world are taking this work home with them.  The process usually goes like this: I check my morning tweets and find out that virtualization company X has released a new version of their product (this typically happens at a conference).  The next step is to go to their site and download the install and administration guides.  Then, curl up next to the fireplace with a hot beverage and be amazed by how much easier my job will become.  OK, that’s not exactly how it typically happens.  Replace fireplace with seat-back tray and hot beverage with nothing and you get the idea.  Personally, I would rather be reading about “Steve Jobs” (barnes and noble, kindle) then another product update novel but this is my life. 

This got me thinking, “how many virtualization products have been released in the last twelve months?”  Let’s see, Citrix released new versions of XenDesktop and XenApp.  Oh, VMware also released a new version of vSphere.  It didn’t take long to realize I would need help counting.  I reached out to my go-to VMware guy, Rene (@vrenenelson), to help me fill in gaps in the VMware product set.  Very quickly, we realized that nearly every major virtualization product has had a revolutionary (as opposed to evolutionary) revamp.

We collaborated to bring you this list.  You may be thinking, “but what about AppSense or Unidesk or [insert your product here]?”  We stuck with the virtualization big three:

Citrix

Microsoft

VMware

That is only the tip of the iceberg. We tried to stick to the major products that administrators have to install. If it is some kind of cloud offering or tech preview we left it off the list. August 2011 should be considered a hallmark month for virtualization due to the perfect storm of major products released.

If you are still looking for your administrator, there’s a good chance they are holed up somewhere going through another admin guide.  Studying like it was the night before the SATs.  Pat your virtualization girl or guy on the back because they are fighting to keep up with an incredibly quick moving industry.

Drop us a line in the comments about what product we scandalously forgot to chart.