Well, PHP is a server-side programming language : server-side? means that the code is interpreted at level of the PHP engine installed on the web server. PHP is open-source (free at no cost). Beeing free of charge makes it very popular and I would say easy to use.
Where can be used php? Everywhere on the web, and I think desktop application can be coded too with the GTK feature but I dont know much about it so I will stop here talking about this. On the other side, for the web application development, you can build dynamic websites, database driven websites (for example MySQL, PostreSQL, etc.), diverse web forms like a contact form which will insert the infos into database system or send details entered in the form to a specified email address, etc.
How do you start the ball rolling? Well, please start reading introductory tutorials on php official website, then get a project in mind and try finding requirements needed for this project. This is the best way to learn : by practicing, not just reading tons of borring tutorials.
Once you got a project in mind, you will definitely need a php engine to be able to execute the code. How do you make this? Simply install a webserver like apache, add php server to this and mysql database engine. You'll find a lot of documentation to help you doing this or there are a lot of ready made packs wich includes, php, apache, phpmyadmin, mysql, etc for example easyphp web server or xampp. They're free and easy to install.
The second thing is to register at specific forums, where you can get support and help from the php gurus, because you really need fast help when errors will occure, and by the way, trust me you'll encounter a lot of "strange" errors at the begining, which later will become just a routine.
And lastly, do not forget to get a nice code editor with code colloring, code auto-completion feature is very important too which will make developing faster. My tip here is to use Zend IDE (the php company) which is the best and includes mentioned features. Also, Adobe Dreamweaver is not bad, you can use that too which is nice for html/css client side scripting. I assumed that you already know html at least, it is very recommended to know it to make things clear in programming.
And, for the begining this should be enough for a novice to begin to learn php programming language at the basic level. Later, with the time and projects which are mandatory for progress, you'll get experience and will advance maybe to OOP (object oriented programming) which is another thing important.
