Complexity and difficulty is subjective. Story points are an estimation of the quantity of work, complexity and uncertainty associated with a task. Though not always more accurate than “time-based” estimates, story points are more flexible as they aren’t tied to individual skill levels and don’t need to be “resized” if something changes. If something does change (a team member goes on vacation or the team’s efficiency increases), you can adjust the number of story points with each sprint.
The poker system assigns points in a democratic fashion (secret votes that are simultaneously revealed). Substantial variations may be discussed or you may simply assign it the most points as Planning Poker suggests if you have a mature team. It’s useful to define and share examples, conditions, and characteristics of story point values so that everyone has a common point of reference.
Most teams use the Fibonacci Sequence (1, 2, 3, 5, 8, 13, 21) as each increment is noticeably different (~60%). Instead of quibbling over whether a task is a 3, 4, or 5, assigning it the highest value still accommodates everyone. It reduces the risk of the task spilling over into the next sprint. If someone finishes all of their tasks early, they can either help others finish their tasks or take a new one from the backlog. Either way, it’s all reflected in team velocity which is helpful in planning future sprints. This doesn’t improve cycle-time per se.