It's a good habit to take off the ones with stems that go into actual buds. They will carry rot into a bud. I don't really see any that are in danger of doing that just yet, but it's at this stage it can happen fast. Keep an eye out for that.