I found it very difficult to delete my Google plus profile picture. I am writing the step by step tutorial to help you to delete your unwanted profile picture or other album particular photos.
1. Login to your Google plus account
2. Click on photos from top menu.
3. Select your albums
4. Select profile photos
5. Select the profile picture you want to delete
6. Click option menu in the bottom left hand corner as shown in the picture.
7. Click Delete photo, its done.
Old delete option is not available now, instead there is new delete button available now in a toolbar just above your picture. Please check the snapshot below for reference. This new change make more easy to delete your unwanted profile and other pictures.