The thesis introduces the new concept of dendroidal set. Dendroidal sets are a generalization of simplicial sets that are particularly suited to the study of operads in the context of homotopy theory. The relation between operads and dendroidal sets is established via the dendroidal nerve functor which assigns to each operad a dendroidal set. This is done in such a way as to generalize the nerve construction of a category as introduced by Grothendieck. The main concept introduced and studied within the theory of dendroidal sets is that of an inner Kan complex. This is a dendroidal set satisfying certain filling conditions that endow the dendroidal set with a very rich structure. This concept can be seen as a generalization of quasi-categories, which are recently beeing studied by Joyal. It is shown that the theory of dendroidal sets lends itself very naturally to the study of up-to-homotopy algebras in the context of monoidal model categories. Our approach provides for a very natural definition of categories enriched in a dendroidal set, and this definition can be iterated. This results, among other things, in a definition of weak n-categories. There is a close relation between dendroidal sets and simplicial sets, foremost when considering Joyal's model structure on simplicial sets. In that context we provide a proof that the inner Kan dendroidal sets form an exponential ideal with respect to certain normal dendroidal sets. This result, when restricted to simplicial sets, provides a new proof for the result of Joyal that quasi-categories form an exponential ideal in simpclicial sets. The general theory of dendroidal sets seems to be of relevance in whenever up-to-homotopy algebras are being studies (thus A_\infty-spaces and A_\infty-categories), as well as in the general theory of operads.