Lack of Capacity
Capacity refers to a person's ability to enter into a legally binding agreement. By contesting the will on the grounds of lack of capacity, you are essentially saying the testator was not "of sound mind" at the time of making the will, and didn't really understand the document being executed. State law requirements for capacity vary, but generally you must show that the testator -- the maker of the will -- did not understand the scope of her estate, the relationships between herself and named or unnamed beneficiaries, and that she either didn't understand the consequences of the distributions in the will or lacked the ability to make a rational decision.
Failure of Formality
Formality refers to the requirements of a valid will as defined by state law. Formalities usually require that a will be in a writing, signed by the testator and at least two witnesses that are not beneficiaries to the estate. However, there are variations in each state: some states require three witnesses and/or notarization of the will, and a few states recognize unwritten handwritten wills or witnessed oral wills in certain circumstances. Formality most likely cannot be contested if a will meets the state's requirements for a self-proving will, which means it is accompanied by the witnesses' notarized affidavits.
Fraud or Undue Influence
A will may be invalidated in totality or in part if the dispositions it contains are the product of fraud or undue influence. Fraud occurs when the testator makes a testamentary disposition based on false information that was intentionally provided for the purpose of misleading the testator. Undue influence is the result of applying coercive measures on the testator such that her own mental faculties are overcome and a disposition is made to a person who would not have otherwise received the property. Application of pressure on the testator can take a variety of forms, including denial of legal counsel and rushing the will-making process. Courts typically have discretion in deciding whether undue influence was present in the formation of a will, and usually require meaningful evidence.
A will can be contested for a variety of other reasons. There may be a mistake in the formality of the will, such as if the testator signed in the wrong place, and the court may either choose to overlook the mistake, invalidate the will or make some compromise. A will can be contested on the grounds that it is not the most current will, or that it does not revoke part of an older will that should remain in force. In some states, a divorce subsequent to the will invalidates all or part of the will so that it can be contested if presented in court. It's also possible that the way particular property is conveyed in the will is not a valid testamentary disposition under the laws of the state. Short of actually contesting the will itself, it may also be possible in some states to file a civil suit against the beneficiary receiving property you believe is rightfully yours.