Moving Files across the network…for 2000 machines


Today I got very happy and to be honest, with enormous sense of job satisfaction.  A student (and a very good friend) of my class of PowerShell 3.0 Fundamentals training had a problem and could solve it using what he learned.

I just did some adjustments to work in asynchronous mode, but he  built the solution from himself.

He works for for a LARGE  IT  Corporation in Redmond and the production, in his division, was  off.

What he needed to do, was move files that had X MB from a path to another across the network. It is a simple GCI and Move-Item you can be thinking.

It is, if was not the case to perform this operation for 2000 machines, without PowerShell Remoting (it is not enabled) and in the short time possible. The production was off.

The code  is pretty simple :

 

Get-Content C:\Server.txt |
ForEach-Object {
    Get-ChildItem \\$_\X$\foo\foo1\ -file |
    Where { ($_.Length/1MB) -gt X } |
    Move-Item  -destination \\FixedPath\foo\LargeFiles
}

 

Now, the BIG problem is to scale this code to 2000 computers. We cannot do it in synchronous  mode.

First there is no time : Production is off and to perform this code sequentially it would take a LOT of time.

I could have wrote a script to use runspaces or background jobs controlling the queue but again.. time is the problem now.

In my class I ALWAYS SAY to my students, write REUSABLE functions and do not reinvent the well, and today one of them saw why.

There is one of my favorites function called Split-Job. You can find it in the best source to PowerShell scripts and functions, the POSHCODE  – POSHCODE SPLIT-JOB

I use this function almost 3 years, in production, and it is AMAZING (and it was wrote for PowerShell 1.0). It creates to you runspaces , with a LOT of parameters to manage them.

Our goal was create 32 rusnpaces and controlling the queue after that.  The final solution was :

Get-Content C:\Server.txt |
Split-Job {
    ForEach-Object {
        Get-ChildItem \\$_\X$\foo\foo1\ -file |
        Where { ($_.Length/1MB) -gt X } |
        Move-Item  -destination “\\FixedPath\foo\LargeFiles”
    }
} -maxpipelines 32

and the job was done, in the minimum time and clean.

Of course , he learned in my class, TEST FIRST USING -WHATIF. He had some dev servers to test , he simulate the problem using 3 servers and  did it, using –WHATIF

We could write the code use runspaces, but what’s the point ?  Already exists a perfect function to do that and probably he solved his problem at least  50% faster than  if we had tried to write something.

What I am trying to say is , IMHO, we need to focus in the solution and solve the problem. I don’t need to prove or show that I can write amazing codes. My job it is provide solutions and solve the problem ASAP.

For that  Just keep reading, learning and always with you eyes opened to the PowerShell Community. You will be impressed on what your friends already have done and they are sharing for free.

You don’t need to  be the best PowerShell coding guy..

You just need to be the BEST in to know WHERE TO FIND your answers

I guess the name for that is “Experience” .   And that certainly applies for life too 🙂

PS – if you still think that you don’t need to learn PowerShell, think about it. Perhaps you can make and be the difference. As my friend did 🙂

About Laerte Junior

Laerte Junior Laerte Junior is a SQL Server specialist and an active member of WW SQL Server and the Windows PowerShell community. He also is a huge Star Wars fan (yes, he has the Darth Vader´s Helmet with the voice changer). He has a passion for DC comics and living the simple life. "May The Force be with all of us"
This entry was posted in Algo que Esqueci de Categorizar. Bookmark the permalink.

2 Responses to Moving Files across the network…for 2000 machines

  1. Pingback: Moving Files across the network…for 2000 machines – Part II | $hell Your Experience !!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s