In a letter to C. Schmidt on 5 August 1890, Marx's fellow-author Friedrich Engels wrote: "As Karl Marx used to say about the French "Marxists" in the 1870s, 'All I know is that I am not a Marxist.'"
I think Marx was referring particularly to the period of the Paris Commune. In 1871 an attempt to overthrow the monarchist French government led to widespread rioting, violence and near civil war. Many leaders of the anti-government movement called themselves Marxists. This annoyed Marx, but it wasn't the only development in "marxism" which annoyed both him and Engels. They lived to see their theories, which were intended to encourage enlightened research and debate, treated like religious doctrines, and often twisted to suit various agendas.